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The  Department  of  Defense  has  recently  declared  that  irregular  warfare  is  as 
strategically  important  as  traditional  warfare.  Unfortunately,  there  is  a  dearth  of  mature 
training  and  analysis  tools  that  can  be  used  to  support  contemporary  military  operations. 
One  popular  wargaming  simulation  is  the  campaign-level  Peace  Support  Operations 
Model  (PSOM).  This  thesis  provides  a  quantitative  analysis  of  PSOM.  The  results  are 
based  on  over  75,000  simulated  runs  of  an  Operation  Iraqi  Freedom  scenario.  The  study 
concludes  with  the  identification  of  the  critical  factors  within  PSOM,  recommended 
potential  uses  for  the  model,  an  accuracy  assessment,  and  an  assessment  of  the  risks 
assumed  by  using  the  model.  Results  indicate  that  the  critical  factors  within  the  model 
are  indicative  of  contemporary  operations.  PSOM  should  be  used  for  its  original 
purpose,  as  a  wargame  to  further  study  the  societal  implications  of  modem  military 
operations.  As  a  wargame,  PSOM  has  strong  potential  as  a  high-level  staff  and  leader 
training  tool  and  as  a  planning  aid  for  course  of  action  development.  Within  the  confines 
of  this  study,  the  model  proved  limited  in  its  ability  to  model  changes  in  force 
capabilities.  Due  to  its  limited  ability  to  model  uncertainties  in  irregular  warfare  without 
the  human-in-the-loop,  or  give  multiple  potential  outcomes,  further  development  and 
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EXECUTIVE  SUMMARY 


This  research  addresses  the  Department  of  Defense’s  (DoD’s)  need  to  accurately 
model  strategic  level  contemporary  military  operations.  In  the  simplest  form,  the  purpose 
is  to  gain  a  basic  understanding  of  the  Peace  Support  Operations  Model  (PSOM).  In 
doing  so  we  provide  a  methodology  which  should  be  incorporated  into  the  VV&A 
(verification,  validation,  and  accreditation)  process  for  complex  combat  models  that 
incorporate  the  Political,  Military,  Economic,  Social,  Infrastructure,  and  Information 
(PMESII)  aspects  of  irregular  warfare.  This  summary  explains  the  need  for  such  a  model 
and  hence  the  methodology,  the  steps  taken  during  the  analysis  of  PSOM,  and  the 
conclusions  and  recommendations  toward  the  future  use  of  PSOM  and  the  study  of 
PMESII  models.  The  following  document  provides  the  DoD  insights  into  the  potential 
use  of  PSOM  as  a  wargame  and  as  an  analytic  tool. 

As  the  United  States  Military’s  focus  shifts  from  conventional  warfare  toward 
irregular  warfare,  interest  has  progressively  grown  in  the  development  of  models  that  can 
simulate  social  behavior  as  it  pertains  to  military  operations.  Populations,  whether 
broken  into  smaller  social  groups,  granulated  into  individuals,  or  studied  as  an  aggregate 
of  social  groups,  are  often  the  determinate  of  success  in  modem  combat.  According  to 
the  most  recent  U.S.  Army  doctrine: 

The  integration  of  civilian  and  military  efforts  is  crucial  to  successful 
COIN  [Counter  Insurgency]  operations.  All  efforts  focus  on  supporting 
the  local  populace  and  the  H[ost]  N[ation]  government.  Political,  social 
and  economic  programs  are  usually  more  valuable  than  conventional 
military  operations  in  addressing  the  root  causes  of  conflict  and 
undermining  an  insurgency.  FM  3-24  Counterinsurgency 

The  military  uses  models  for  course  of  action  analysis,  training  and  rehearsal,  and 
evaluation  for  acquisition.  If  these  models  are  not  indicative  of  contemporary  operations, 
they  are  not  only  lacking  in  utility,  they  are  potentially  harmful.  Therefore,  the  military’s 
interest  in  modeling  social  cognition  has  grown  out  of  necessity.  To  date  there  has  not 
been  a  validated  model  designed  for  irregular  warfare  that  covers  the  instruments  of 
national  power:  Diplomatic,  Information,  Military,  Economic  (DIME)  or  the  Political, 


Military,  Economic,  Social,  Infrastructure,  and  Information  (PMESII)  indicators  on 
which  progress  in  irregular  warfare  is  based.  Figure  1  illustrates  the  complex  causal 
relationship  between  DIME  and  PMESII  factors.  As  one  can  imagine,  the  development 
of  such  a  model  is  not  trivial.  According  to  the  Defense  Modeling  and  Simulation 
Analysis  Committee,  the  data  to  instantiate  such  a  model  is  either  nonexistent  or  woefully 
inaccurate,  and  the  validation  process  of  such  a  model  would  have  to  be  completely 
rethought. 
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Figure  1 .  The  complexity  of  the  PMESII  Environment  [From  (Allen  2004)] 


One  new  model  that  addresses  the  operational  focus  on  the  population  is  the 
Peace  Support  Operations  Model  (PSOM).  PSOM  is  a  campaign  level,  simulation  based, 
human-in-the-loop  wargame  which  portrays  the  civilian  populace  as  agents  within  the 
model.  The  model’s  metrics  are  both  quantitative  and  qualitative  in  nature,  providing 
results  which  are  analogous  with  irregular  warfare.  Many  of  the  Measures  of 
Effectiveness  offered  by  PSOM,  such  as  attrition,  are  easily  understood.  However,  much 
of  the  output  is  a  cultural  representation  of  the  uncertain  effects  military  and  political 
actions  have  on  the  population. 
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This  analysis  defines  a  logical  methodology  to  assess  PSOM  as  a  potential  tool  to: 

•  Quantitatively  measure  the  limitations  and  constraints  of  PSOM 
and,  more  importantly,  identify  the  appropriate  context  for 
interpreting  PSOM  results. 

•  Assess  the  accuracy  of  PSOM  in  regards  to  current  doctrine. 

•  Make  recommendations  toward  the  potential  use  of  PSOM. 

•  Define  a  methodology  for  the  much  needed  VV&A  process  of  PMESII 
models. 

In  order  to  address  the  fore  mentioned  issues,  this  study  follows  the  data  farming 
process  (Home  2004).  Our  implemented  data  farming  process  entails  a  four  step 
procedure:  Define  factors  of  interest,  create  a  design  of  experiment  (DOE),  ran  the 
simulation  experiments  in  parallel  on  a  computing  cluster,  and  conduct  data  mining.  By 
leveraging  thoughtful  experimental  design  with  powerful  computation  capabilities  we  are 
able  to  change  over  100  parameters  within  PSOM  while  conducting  over  50,000 
simulations  of  operations  in  the  current  Multi-National  Forces  Iraqi  Area  of 
Responsibility.  This  immense  number  of  simulation  runs  allows  us  to  quantifiably 
analyze  over  5,000  design  points  consisting  of  well  over  500,000  data  files  covering  a 
large  portion  of  the  model’s  response  space.  By  covering  such  a  vast  space  of  possible 
outcomes,  we  are  able  to  use  a  variety  of  methods  to  investigate  the  model’s  behavior. 

The  analysis  of  this  data  set  focused  on  three  doctrinally  essential  measures  of 
effectiveness  (MOEs)  provided  by  PSOM  in  a  current  Iraq  based  scenario.  These 
measures  are  the  Sunni  Population’s  consent  toward  the  Coalition  forces,  the  Sunni 
Population’s  consent  toward  the  Iraqi  Government,  and  the  security  level  of  the  nation. 
The  results  of  our  research  are  telling. 

PSOM  provides  players  a  tremendous  amount  of  flexibility  in  choosing  their 
operational  tasks  (stances).  Our  analysis  shows  that  the  stances  players  (factions)  choose 
have  a  significant  effect  on  both  the  consent  and  security  of  the  population,  which  is  what 
one  would  expect  based  on  current  doctrine.  There  are  also  intuitive  interactions  between 
the  stances  of  multiple  factions.  The  rules  of  engagement  status  and  risk  level  of  a  unit  in 
conjunction  with  the  unit’s  stance  also  prove  important  choices  throughout  the  game. 
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For  example,  a  violent  ROE  in  conjunction  with  an  aggressive  stance  will  result  in  a  loss 
of  consent,  especially  if  another  faction  proves  more  cautious  and  focused  on  the 
population. 

We  found  the  consent  metric  to  be  more  intricate  than  the  security  metric.  It 
seems  consent  is  difficult,  but  not  impossible,  to  increase  toward  both  the  coalition  and 
the  Iraqi  government.  It  is  arguable  that  this  difficulty  is  actually  a  strength  of  the  game. 
Changing  the  opinion  of  a  society  is  a  particularly  difficult  task;  just  as  in  PSOM,  it  can 
be  done,  but  not  without  deliberate  effort. 

The  initial  conditions  and  assumptions  made  in  scenario  design  are  of  particular 
importance.  The  simulation’s  results  can  be  dominated  by  some  of  these  assumptions. 
For  example,  the  underlying  assumptions  that  are  used  to  create  the  population  are 
absolutely  critical  in  the  determination  of  consent  in  the  game.  In  accordance  with 
doctrine  and  the  PSOM  developers,  the  outcomes  of  PSOM  are  especially  population 
centric.  Because  of  this,  if  underlying  assumptions  about  the  population  are  not  strongly 
supported,  the  game  should  be  played  multiple  times,  changing  population  assumptions 
to  ensure  a  considerable  range  of  possible  outcomes  is  covered. 

The  game  is  not  very  sensitive  to  many  of  the  non-scenario  specific  parameter 
settings.  Of  particular  focus  within  this  study  are  those  of  unit  capabilities.  Therefore, 
we  recommend  PSOM  should  not  be  used  as  a  decision  tool  for  equipment  manning  or 
force  manning  until  this  aspect  of  the  game  is  studied  further.  Those  settings  to  which  we 
found  the  game  to  be  sensitive  are  mentioned  within  the  thesis. 

The  model  should  be  used  for  its  original  purpose,  as  a  wargame  to  further  study 
the  societal  implications  of  modem  military  operations.  As  a  wargame,  PSOM  has  strong 
potential  as  a  high-level  staff  and  leader  training  tool  and  as  a  planning  aid  for  course  of 
action  development  for  stability  operations.  Within  the  confines  of  this  study,  the  model 
proved  limited  in  its  ability  to  model  changes  in  force  capabilities.  Also,  due  to  its 
inability  to  model  uncertainties  in  irregular  warfare,  or  give  multiple  potential  outcomes 
further  development  and  analysis  is  required  before  the  model  is  used  for  large  scale 
analysis. 
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I. 


INTRODUCTION 


At  an  earlier  time,  a  commander  could  be  certain  that  a  future  war  would 
resemble  past  and  present  ones.  This  enabled  him  to  analyze  appropriate 
tactics  from  past  and  present.  The  troop  commander  of  today  no  longer 
has  this  possibility.  He  knows  only  that  whoever  fails  to  adapt  the 
experiences  of  the  last  war  will  surely  lose  the  next  one. 

German  Gen.  Franz  Uhle-Wettler  (1985) 

A.  OVERVIEW 

As  the  focus  of  the  United  States  military  shifts  from  conventional  warfare  toward 
irregular  warfare,  interest  has  progressively  grown  in  the  development  of  models  that  can 
simulate  social  behavior  as  it  pertains  to  military  operations.  The  contemporary 
operating  environment,  as  reflected  in  Iraq  and  Afghanistan,  shows  the  critical  role  the 
population  plays  in  modern  combat.  Populations,  whether  broken  into  smaller  social 
groups,  granulated  into  individuals,  or  studied  as  an  aggregate  of  social  groups,  are  often 
the  determinate  of  success  in  modem  combat.  Therefore,  the  military’s  interest  in 
modeling  social  cognition  has  grown  out  of  necessity.  The  military  uses  models  for 
course  of  action  analysis,  training  and  rehearsal,  and  evaluation  for  acquisition.  If  these 
models  are  not  indicative  of  contemporary  operations,  they  are  not  only  lacking  in  utility, 
but  are  potentially  harmful. 

One  new  model  that  combines  conventional  warfare  with  the  modem  focus  on  the 
population  is  the  Peace  Support  Operations  Model  (PSOM).  PSOM  is  a  simulation-based 
wargame,  that  portrays  the  populace  and  displays  the  effects  military  and  political  actions 
have  on  the  population  and  its  emersion  of  social  organizations.  During  preliminary  use 
of  PSOM  at  the  Joint  Staff  Warfighting  Analysis  Division  and  the  Office  of  the  Secretary 
of  Defense,  PSOM  has  shown  potential  as  an  analytic  and  training  tool;  however,  to  date 
the  model  has  not  been  taken  through  any  sort  of  verification,  validation  and  accreditation 
(VV&A)  process.  VV&A  is  an  important  prerequisite,  ensuring  models  with  flaws  or 
biases  to  not  become  mainstream  tools.  The  lack  of  appropriate  vetting  can  prove 
detrimental  due  to  the  tremendous  risk  inherited  from  using  a  model  which  may  provide 
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inconsistent  or  inaccurate  results.  This  study  quantifiably  analyzes  PSOM  using  design 
of  experiments  and  data  fanning  to  quantitatively  measure  the  limitations  and  constraints 
of  the  model. 

B.  BACKGROUND 

Warfare  changes  with  society.  Over  the  past  fifty  years  we  have  seen  a  shift  from 
maneuver  warfare  where  nonlinear  tactics  could  determine  victory  (German  blitzkrieg), 
to  irregular  warfare  where  the  objective  is  the  support  of  a  population  (Mao  Tse-tung  and 
the  Protracted  People's  War).  Irregular  warfare  is  a  violent  struggle  among  state  and  non¬ 
state  actors  for  legitimacy  and  influence  over  the  relevant  population  (Department  of 
Defense  [DoD]  Directive  3000.07,  2007).  The  Department  of  Defense’s  policy  is  to 
“Recognize  that  IW  [irregular  warfare]  is  as  strategically  important  as  traditional 
warfare”  (DoD,  2007).  Irregular  warfare  contains  many  subsets,  including 
unconventional  warfare,  foreign  internal  defense,  counter  insurgency  (COIN), 
counterterrorism,  and  stability  operations. 

The  United  States  military  continues  to  transform  to  meet  the  current  and 
upcoming  challenges  of  national  security.  Although  the  United  State  military  must 
remain  able  to  defend  the  country  and  defeat  a  robust  enemy  in  a  conventional  war,  the 
military  now  must  conduct  the  complete  taxonomy  of  irregular  warfare  to  ensure 
America’s  security.  In  doing  so,  the  military  focus  is  split  between  the  destruction  of 
enemy  armies  and  the  development  of  “indigenous  capacity  for  securing  essential 
services,  a  viable  market  economy,  rule  of  law,  democratic  institutions,  and  robust  civil 
society”  (DoD  Directive  3000.05,  2005);  the  latter  obviously  being  the  far  more  difficult 
to  plan.  As  C.E.  Callwell  states  in  Small  Wars  (1906): 

But  when  there  is  no  king  to  conquer,  no  capital  to  seize,  no  organized 
army  to  overthrow,  and  when  there  are  no  celebrated  strongholds  to 
capture,  and  no  great  centers  of  population  to  occupy,  the  objective  is  not 
so  easy  to  select,  (p.  40) 
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Colonel  Callwell  continues  by  explaining  how  an  attacking  force  must  destroy 
and  deprive  the  defending  population  using  techniques  that  are  inhumane  and  not 
accepted  under  today’s  laws  of  war.  Contemporary  doctrine  enforces  the  mitigation  of 
civilian  casualties  and  collateral  damage. 

The  role  of  the  military  has  become  extremely  complex,  interwoven  with  other 
governmental  and  non-governmental  agencies.  Current  war  spans  political,  economic, 
social  and  military  aspects  of  humanity  (Hammes,  2008).  The  military  has  published 
new  doctrine  and  leaders  have  been  quick  to  change  their  mindset.  As  young  men  and 
women  gain  experience  in  this  emerging  warfare,  it  has  become  the  norm  rather  than  the 
exception  for  warfighters  to  leverage  the  political,  economic,  social,  and  military  aspects 
of  war.  The  result  is  a  military  composed  of  people  ready  to  meet  the  irregular  warfare 
challenges  and  accomplish  the  mission  in  stride.  This  trend  will  continue  as  junior 
leaders  become  senior  leaders  with  a  wealth  of  modern-day  experience  and  knowledge 
which  will  in  turn  become  wisdom. 

As  the  paradigm  of  the  military’s  role  changes,  the  plans,  training,  and  force 
structure  must  be  dynamic  as  well.  Unfortunately,  a  key  part  of  the  planning  process, 
training  structure,  and  military  decision  making  process  has  yet  to  be  transformed.  This 
key  ingredient  is  the  modeling  and  simulation  of  war.  Whether  manifested  in  a  wargame, 
simulation,  or  experiment,  simulation  modeling  has  become  a  critical  part  of  the  decision 
making  process.  An  example  is  the  critical  role  that  modeling  takes  in  DoD’s 
development  of  the  Analytic  Agenda  (Stevens,  2003). 

According  to  the  Committee  on  Modeling  and  Simulation  for  Defense 
Transformation,  “DoD  needs  MS&A  appropriate  to  complex  dynamic,  adaptive  systems 
because  such  systems  pervade  military  combat,  other  aspects  of  military  operations,  and 
other  political,  military,  economic,  social,  infrastructure,  and  information  phenomena  of 
interest”  (Committee  on  Modeling  and  Simulation  for  Defense  Transformation,  2006). 
DoD’s  Transformation  Planning  Guidance  (2003)  states  “a  new  generation  of  M&S  is 
needed  to  support  concept  development  linking  together  many  types  of  simulations  from 
aggregate  and  detailed  computer  models  to  simulators  and  man  in  the  loop  hardware 
components.” 
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The  Department  of  Defense  defines  validation  as  the  process  of  determining  the 
degree  to  which  a  model,  simulation,  or  federation  of  models  and  simulations,  and  their 
associated  data  are  accurate  representations  of  the  real  world  from  the  perspective  of  the 
intended  use(s)  (DoD,  2008).  To  date  there  has  not  been  a  validated  model  designed  for 
irregular  warfare  that  covers  the  instruments  of  national  power,  Diplomatic,  Information, 
Military,  Economic  (DIME)  or  the  Political,  Military,  Economic,  Social,  Infrastructure, 
and  Information  (PMESII)  indicators  that  progress  in  irregular  warfare  is  based.  Figure  2 
illustrates  the  complex  causal  relationship  between  DIME  and  PMESII  factors.  As  one 
can  imagine,  the  development  of  such  a  model  is  not  trivial.  According  to  the  Defense 
Modeling  and  Simulation  Analysis  Committee,  the  data  to  instantiate  such  a  model  is 
either  nonexistent  or  woefully  inaccurate,  and  the  validation  process  of  such  a  model 
would  have  to  be  completely  rethought  (Committee  on  Modeling  and  Simulation  for 
Defense  Transformation,  2006). 
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Figure  2.  The  Complexity  of  the  PMESII  Environment  [From  (Allen,  2004)]  [Best 
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Gropman  (1986)  states  a  popular  position  in  the  development  of  mathematical 
models  pertaining  to  human  social  interaction  as: 

Psychological  dimensions:  their  uncertainty,  their  variety,  their 
inconsistency  and  their  lack  of  utility  in  modeling  future  conflict  should 
make  one  reluctant  to  expend  resources  pursuing  them,  leaving  the 
community  the  time  and  money  to  quantify  better  the  quantifiable,  (p.  16) 

When  modeling  warfare,  especially  on  a  macro  level,  much  of  the  information 
about  human  behavior  is  either  assumed  or  discarded,  often  resulting  in  a  gross 
simplification  of  the  situation  we  wish  to  model  (Perla,  1990,  p.  276).  This  problem  is 
amplified  when  attempting  to  model  irregular  warfare.  In  FM  3-07,  the  newest  Army 
Manual  on  Stability  Operations,  LTG  Caldwell  states: 

The  lines  separating  war  and  peace,  enemy  and  friend  have  blurred  and  no 
longer  conform  to  the  clear  delineations  we  once  knew.  At  the  same  time, 
emerging  drivers  of  conflict  and  instability  are  combining  with  rapid 
cultural,  social,  and  technological  change  to  further  complicate  our 
understanding  of  the  global  security  environment.  (Army,  2008) 

This  statement  alone  depicts  the  modeling  of  irregular  warfare  as  a  great  problem  to  even 
comprehend  let  alone  attack. 

Our  military  has  become  dependent  on  models.  Traditionally  the  military  has 
used  computational  models  for  analysis  and  forecasting  for  planning,  simulation  for 
training  rehearsal,  and  design  and  evaluation  for  acquisition  (Committee  on 
Organizational  Modeling,  2008,  p.  23).  However,  we  are  now  fighting  a  war  that  many 
deem  extremely  difficult  to  model,  much  less  model  in  a  way  that  would  fit  traditional 
validation  techniques.  Due  to  the  dependence  on  models  and  simulation,  the  DoD  has 
attacked  this  problem  with  vigor,  and  the  modeling  community  has  answered  the  call  to 
develop  what  are  now  being  called  PMESII  models. 

There  are  many  models  being  developed  to  meet  the  need  to  understand  the 
contemporary  battlefield.  Currently  these  models  fall  into  multiple  categories  such  as 
agent  based,  system  dynamic,  or  analytic.  These  models  differ  in  their  underlying 
designs  used  to  generate  outputs.  However,  they  are  similar  in  that  they  are  a  step  in  the 
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direction  of  modeling  the  asymmetric  battlefield  where  “civilians  are  the  targets, 
objectives  to  be  won,  as  much  as  an  opposing  force”  (Smith,  2007).  One  such  model  is 
the  Peace  Support  Operation  Model  v2  (PSOM  2). 

PSOM  was  developed  by  the  United  Kingdom  (UK)  to  first  study  and  then 
understand  stabilization  operations.  Model  development  began  in  2004  at  the  Defense 
Science  and  Technology  Laboratory,  United  Kingdom.  The  model  is  built  by  an  allied 
institution  which  has  already  encouraged  inter-agency  use  of  the  simulation.  The  model 
is  a  time  stepped  human-in-the-loop  semi-automated  campaign  level  wargame.  PSOM 
uses  over  eighty  algorithms,  which  are  primarily  deterministic  with  a  number  of 
stochastic  elements,  to  take  into  account  the  DIME  interventions  and  PMESII  indicators 
in  determining  outcomes  of  friendly  unit  actions  (Body,  2008).  These  outcomes  are 
representative  of  required  results  from  irregular  warfare  where  seized  terrain  and 
causality  counts  cannot  determine  a  victor  or  even  positive  results. 

A  critical  underlying  assumption  of  PSOM  is  that  current  UK  and  U.S.  doctrine 
represent  the  best  strategy  in  Peace  Support  Operations.  This  assumption  is  difficult  to 
validate  due  to  the  changing  nature  of  irregular  warfare,  and  doctrine  is  being  developed 
and  improved  on  a  regular  basis.  However,  the  developers  of  the  model  support  this 
assumption  stating  that  the  contemporary  environment  is  being  debated  and  that  PSOM  is 
“an  endorsed  understanding  of  the  Contemporary  Operating  Environment”  (Body,  2008). 
This  statement  is  not  meant  to  be  a  catch-all  for  every  assumption  in  which  the  model  is 
based,  but  rather  a  candid  explanation  of  those  areas  of  “deep  uncertainty,”  which  can 
derive  from  social  modeling. 

The  PSOM  2  model  allows  for  the  interaction  of  multiple  factions  amongst 
themselves  and  the  population.  This  is  key  because  the  model  assumes  that  the 
Contemporary  Operating  Environment  (COE)  is  one  in  which  the  population  is  the  point 
of  emphasis.  The  modelers  understand  that  irregular  warfare  uses  a  range  of  approaches 
including  political,  military,  economic,  and  social  to  persuade  the  enemy  leadership  that 
their  strategic  goals  are  either  unachievable  or  not  cost  effective.  Therefore,  a 
tremendous  emphasis  in  PSOM  is  its  ability  to  represent  both  the  political  will  of  the 

population  and  the  changes  both  friendly  and  enemy  forces  have  on  this  will.  In  order  to 
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model  these  interactions,  PSOM  allows  each  player  or  faction  to  choose  a  stance  for  each 
time  step.  The  stances  are  similar  to  current  doctrinal  operations  and  specific  tactical 
tasks  and  are  directed  toward  other  factions  or  the  population.  For  example,  a  maneuver 
battalion  in  Iraq  could  have  the  stance  of  secure  being  directed  at  the  Sunni  Population. 

PSOM  gives  multiple  measures  of  effectiveness  (MOEs)  and  metrics  that  can  be 
used  to  determine  success  or  lack  thereof.  The  model  is  analogous  with  U.S.  doctrine  in 
that  political  legitimacy  is  a  key  outcome  of  stability  operations.  Legitimacy  is  assessed 
by  the  indicators  of  security,  consent,  rule  of  law,  and  provision  of  essential  services.  The 
primary  MOEs  presented  by  PSOM  are  security,  consent,  stability  and  fear.  These  MOEs 
coincide  with  the  strategic  framework  described  in  FM  3-07,  which  states  the  end  state 
conditions  for  a  stability  operation  are  a  safe  and  secure  environment,  established  rule  of 
law,  social  well  being,  stable  governance  and  a  sustainable  economy.  Although  economy 
is  not  a  stated  MOE  in  PSOM,  economic  conditions  play  a  role  in  all  the  aforementioned 
MOEs  and  multiple  economic  metrics  are  available. 

For  all  of  its  strengths,  to  date  this  model  has  not  been  through  any  particular 
VV&A  process.  The  J-8  Warfighting  Analysis  Division  along  with  the  Office  of 
Security  of  Defense  Simulation  Analysis  Center  (OSD-SAC)  have  used  PSOM  in  an 
exploratory  manner  with  debatable  results.  Currently,  the  staff  at  DSTL  are  conducting  a 
large-scale  multiple  course  of  action  wargame  to  test  PSOM.  However,  PSOM  has  not 
been  put  through  the  rigors  of  the  DoD’s  Modeling  and  Simulation  accreditation  process. 

C.  RESEARCH  QUESTIONS 

The  intent  of  this  research  is  to  conduct  a  quantitative  analysis  of  the  Peace 
Support  Operations  Model.  This  analysis  is  not  meant  to  be  a  complete  W&A  process. 
However,  the  following  issues  are  addressed: 

•  Identify  the  factors  which  most  dramatically  change  PSOMs  output. 
Identifying  the  critical  factors  quantitatively  measures  the  limitations  and 
constraints  of  PSOM,  and,  more  importantly,  identify  the  appropriate 
context  for  interpreting  PSOM  results. 

•  Attempt  to  assess  the  accuracy  of  the  Peace  Support  Operation  Model. 


7 


•  Make  recommendations  toward  the  potential  use  of  the  Peace  Support 
Operation  Model. 

•  Conduct  a  risk  analysis  of  the  Peace  Support  Operation  Model. 

D.  BENEFITS  OF  STUDY 

This  study  provides  the  Department  of  Defense  a  thorough  analysis  of  PSOM.  In 
doing  so  it  aids  decision  makers  in  selecting  potential  uses  of  PSOM  as  a  wargame  and 
analytic  tool.  Concurrently,  we  have  developed  an  analytic  procedure  for  evaluating 
complex  PMESII  models.  This  procedure  provides  analysts  a  reference  in  the  validation 
and  verification  of  models  that  take  into  account  the  importance  of  societal  implications. 
This  study  demonstrates  that  the  use  of  large  scale  data  farming,  combined  with 
thoughtful  design  of  experiments,  can  provide  useful  insight  into  the  complexities  of 
social  modeling. 

E.  METHODOLOGY 

This  thesis  uses  quantitative  analysis  to  explore  the  capabilities  of  PSOM. 
Because  the  parameter  space  in  a  campaign  level  model  such  as  PSOM  is  quite  large,  the 
study  is  based  on  the  leveraging  of  high  performance  computing  and  efficient  design  of 
experiments  to  run  the  model  many  times.  This  process  allows  for  the  exploration  of  a 
very  large  parameter  space  in  a  limited  amount  of  time  (Barry  and  Koelher,  2004). 
Efficient  design  of  experiments  and  statistical  analysis  permits  us  to  determine  which 
parameters  and  interactions  are  significant  in  PSOM  and  what  the  corresponding 
responses  are  to  a  particular  set  of  parameters.  Once  the  significant  factors  are  identified, 
further  exploration  into  particular  sets  of  parameters  permits  analysis  of  the  response 
accuracy  in  accordance  with  current  doctrine. 

The  scenario  used  to  test  the  model  is  the  ongoing  war  in  Iraq  as  of  2004.  This 

model  was  developed  by  DSTL  in  2008  and  has  been  vetted  through  multiple  U.S.  Army 

officers  who  served  in  the  Iraqi  theater  during  this  period  of  time.  The  design  of 

experiments  focus  on  the  underlying  assumptions  about  the  Iraqi  population,  the 

capabilities  and  attributes  of  coalition  and  insurgent  forces,  the  operational  courses  of 

action  taken  by  coalition  forces,  and  the  systematic  settings  of  PSOM.  The  responses 
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analyzed  are  primarily  the  changes  in  security  in  the  nation  and  the  population’s  consent 
towards  its  own  government  and  coalition  forces  (when  needed,  other  outputs  are  taken 
into  account).  The  resulting  statistical  analysis  of  the  simulated  data  is  then  used  to  gain 
insight  into  the  vast  space  of  possible  PSOM  inputs  and  their  corresponding  outputs. 
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II.  PSOM  BACKGROUND 


The  modeling  of  cognition  and  action  by  individuals  and  groups  is  quite 
possibly  the  most  difficult  task  humans  have  yet  undertaken. 

(Committee  on  Organizational  Modeling,  2008,  p.  20) 


A.  BACKGROUND 

This  next  chapter  is  a  combination  of  first  person  user  experience  with  the  model, 
the  2008  PSOM  2  Functional  Specifications,  and  the  2008  PSOM  Philosophy.  The 
purpose  of  this  chapter  is  to  provide  the  reader  a  basic  understanding  of  the  PSOM  model 
and  software.  This  is  not  meant  to  be  a  standalone  user’s  manual,  but  rather  an 
introduction  to  the  framework  of  the  model.  By  understanding  many  of  the 
specifications,  the  data  required,  the  setup  of  the  model,  and  the  execution  of  the 
simulation  the  reader  can  see  not  only  what  the  model  is  intended  to  do,  but  how  it 
accomplishes  it. 

The  Peace  Support  Operations  Model  is  a  human-in-the-loop,  time  stepped  semi- 
automated  wargame  (Parkman,  2008).  The  definition  of  a  wargame  is  a  model  or 
simulation  not  involving  actual  military  forces,  in  which  the  flow  of  events  is  affected  by, 
and  in  turn  affects,  decisions  made  during  the  course  of  those  events  by  players 
representing  opposing  sides  (Perla,  1990,  p.  274).  PSOM  is  a  campaign  level  model  that 
represents  irregular  warfare  at  the  Policy,  Strategic  and  Operational  Levels  (Body,  2008). 
It  was  designed  by  and  is  still  in  active  development  at  the  Defense  Science  and 
Technology  Lab  of  the  UK  Ministry  of  Defense  in  order  to  test  policy  guidance  and 
provide  campaign  context  for  lower  level  modeling.  PSOM  2  models  irregular  warfare  as 
defined  by  the  Department  of  Defense.  Because  the  results  of  an  irregular  warfare 
campaign  rest  on  the  will  of  the  population,  PSOM  shows  the  causal  effects  of  the 
players’  actions  primarily  through  their  effect  on  the  population.  The  model  is  analytical 
and  based  on  the  assumption  that  current  U.S.  and  UK  irregular  warfare  doctrine 
represent  the  solutions  to  irregular  warfare  (Body,  2008).  The  designers  acknowledge  the 
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significance  of  this  assumption,  but  also  accept  that  there  is  no  universally  accepted 
validated  answer  to  irregular  warfare.  So,  until  combat  models  can  be  tied  into  an  agreed 
upon  anthropological  model,  DSTL  has  moved  forward  with  PSOM. 

PSOM  is  written  in  Visual  Basic  and  is  non-proprietary.  For  this  study  and  future 
studies,  analysts  are  not  just  allowed  to  “look  under  the  hood”  but  encouraged  to  do  so. 
PSOM  is  built  from  a  series  of  algorithms  and  sub-models  which  are  interwoven  to 
represent  the  DIMEFIL  and  PMESII  aspects  of  irregular  warfare.  In  this  aspect  it  is 
possible  to  classify  PSOM  as  a  system  dynamics  model.  In  addition,  the  emersion  of 
social  groups  within  the  civilian  population  is  portrayed  by  independent  agents  whose 
actions  and  attitudes  are  determined  by  simple  decision  rules,  thus  giving  PSOM  a 
pseudo  agent  based  characterization.  One  key  aspect  is  PSOM  supports  the  interaction 
of  multiple  organizations  that  are  representative  of  IW.  PSOM  uses  over  eighty 
algorithms  to  drive  the  unit  interaction  and  state  functionality  systems.  These 
interactions  and  functionality  systems  allow  for  3rd  and  4th  level  actors,  such  as  non¬ 
governmental  organizations,  to  have  effects  on  outcomes. 

B.  SPECIFICATIONS  AND  CAPABILITIES  OF  PSOM 

The  PSOM  wargame  is  designed  with  a  two-level  hierarchy:  the  high  level  game 
(HLG)  and  the  operational  game.  The  high  level  game  is  designed  to  simulate  the 
political  and  strategic  levels  of  conflict.  This  is  where  relationships  between  factions  are 
represented  and  international  cooperation  is  accounted  for.  The  high  level  game  is  very 
much  the  “grey  beard”  game. 

The  operational  level  game  is  just  that,  operational.  It  is  representative  of  the 
campaigns  and  actions  required  to  support  the  strategic  objectives.  This  resolution  is  the 
level  at  which  Brigade  Combat  Teams,  terrorist  organizations,  and  local  government 
organizations  are  played.  By  creating  this  ontology  the  developers  have  attempted  to 
represent  the  importance  of  the  political  and  strategic  influences  with  regards  to  the 
operational  levels  of  irregular  warfare. 

The  PSOM  2  wargame  model  can  be  split  into  four  categories:  scenario  design, 

underlying  data  and  settings,  game  play,  and  results. 
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C.  SCENARIO  DESIGN 

In  the  development  of  social  models  the  represented  environment  is  dynamic  and 
the  model’s  structure  is  often  derived  from  logical,  rather  than  only  mathematical, 
specifications  (Gilbert,  2005,  p.  15).  PSOM’s  scenario  design  accounts  for  this  dynamic 
behavior.  The  scenario  design  is  the  point  at  which  the  modeler  develops  the  specific 
initial  conflict  setting.  In  a  basic  wargaming  analogy,  this  is  where  we  pick  the  layout  of 
the  game  board,  the  attributes  of  the  pieces,  and  the  mindset  of  the  population.  Many  of 
the  basic  assumptions  about  the  social  model  are  inputted  into  the  scenario  design.  An 
abridged  version  of  the  basic  inputs  required  for  the  scenario  follows. 

1.  Factions 

A  faction  is  defined  as  any  political  entity  that  has  an  effect  on  the  scenario. 
These  are  the  key  players  in  the  game.  Faction  examples  are  military  organizations,  the 
host  nation  government,  NGO’s,  and  terrorist  organizations.  A  significant  characteristic 
in  the  makeup  of  a  faction  is  its  ideology.  Factions  have  ideologies  represented  by  the 
Nolan  Chart,  which  uses  scores  for  economic  and  personal  freedom  to  determine  the 
faction’s  political  and  social  inclination.  Figure  3  shows  the  Nolan  Chart  and  gives  the 
explanation  from  the  PSOM  Manual.  Other  inputs  for  each  faction  are  type  of  unit  or 
organization  (there  are  many  pre-set  unit  types),  size  in  number  of  people,  expectations, 
wealth  level,  etc. 

The  Nolan  chart  is  used  to  determine  a  faction’s  political  views.  A  faction  is 
assigned  a  number  based  on  the  importance  of  political  freedom  versus  economic 
freedom.  Where  prior  tools  determined  if  an  individual’s  views  were  to  the  right  or  left 
on  a  line,  the  Nolan  chart  is  a  plane.  The  Nolan  chart  was  developed  by  the  libertarian 
party  and  its  role  in  PSOM  has  drawn  criticism  due  to  its  bias  and  lack  of  scientific 
support  (Tumley,  2008). 
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Figure  3.  An  Example  of  a  Nolan  Chart  Taken  from  the  PSOM  User  Manual 

(Draft) 


2.  Ethnic  Groups 

This  category  represents  the  attributes  of  groups  of  similar  people  within  the 
population.  For  the  model,  ethnic  groups  can  be  used  for  more  than  just  ethnicities,  but 
also  social  views  (Parkman,  2008).  Ethnic  groups  also  use  the  Nolan  Chart  to  determine 
political  ideology.  However,  ethnic  groups  are  assigned  marginal  gains  coefficients. 
Marginal  gains  coefficients  represent  the  importance  of  a  particular  good  or  service  (to 
include  security)  to  the  corresponding  ethnicity.  These  marginal  gain  coefficients  are 
used  to  determine  consent  in  the  operational  game.  Other  inputs  into  the  ethnic  group 
category  are  starting  values  of  population  size,  age  groups,  etc.  As  the  aggregate  of 
ethnic  groups  create  a  population  agent,  the  assumptions  that  create  an  ethnic  group 
should  prove  to  be  of  the  utmost  importance. 

3.  Nationalities 

Every  faction  has  a  nationality.  This  attribute  allows  the  game  to  look  at  the 
homeland  consent  based  on  activities  particular  to  the  individual  nations. 

4.  Map 

The  map  plays  an  important  role  in  the  PSOM.  The  user  downloads  the  map  of 
the  area  of  concern.  Then  the  map  is  divided  by  grid  squares  representing  a  user 
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determined  geographic  size.  The  user  then  gives  each  square  particular  attributes  based 
on  physical  terrain  data,  population  level,  infrastructure  level,  and  human  capital. 


5.  Population  Agents 

By  using  the  input  of  map  data  and  ethnic  group  data,  population  agents  are 
formed.  These  are  “Semi  Autonomous  groups  of  ‘similar’  people  of  a  defined  size  based 
on  proximity.  An  agent  may  make  a  number  of  decisions  based  on  the  conditions  and 
environment  surrounding  it”  (Parkman,  2008,  p.  41).  The  population  agents  gain  the 
attributes  of  their  respective  ethnic  group  with  a  predetermined  user  defined  variance. 

6.  Relationships 

PSOM  allows  the  user  to  define  overarching  relationships  between  factions. 
These  relationships  can  be  updated  throughout  the  game,  but  are  initialized  in  the 
scenario  design.  The  relationships  are  depicted  below  in  Table  1. 


Combative  Relationship 

Intelligence  Relationship 

Shooting 

Sharing 

Shooting 

Not  Sharing 

Not  Shooting 

Sharing 

Not  Shooting 

Not  Sharing 

Table  1.  Explanation  of  Relationships  in  PSOM 


7.  Units 

These  are  the  conventional  military  forces,  nonconventional  military  forces, 
insurgent  or  terrorist  forces,  and  governmental  and  non-governmental  agencies.  The  user 
can  give  units  particular  leadership  values,  experience  values,  stances,  etc.  This  is  where 
you  build  the  hierarchy  of  units  and  commanders  and  assign  the  units  to  their  initial 
location.  This  also  allows  for  the  building  of  battle  groups. 
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D.  DATA  AND  SETTINGS 


The  underlying  data  and  settings  for  the  model  establish  the  basic  characteristics 
and  assumptions  that  can  be  generalized  over  multiple  scenarios.  These  characteristics 
are  predominately  independent  of  the  scenario,  but  can  be  updated  to  reflect  scenario 
specific  requirements. 

1.  Unit  Abilities 


Under  data  and  settings,  the  user  is  able  to  prescribe  unit  abilities.  This  category 
allows  the  user  to  assign  both  descriptive  and  subjective  values  to  multiple  attributes  for 
each  unit  type.  These  values  determine  the  unit’s  ability  to  accomplish  particular  tasks. 
As  above,  a  unit  is  any  organization  that  participates  in  the  scenario.  Figure  4  shows  that 
the  unit  abilities  sub  category  can  be  separated  into  military  values  and  reconstruction 
values.  The  displayed  unit  is  a  health  team,  and  therefore  has  relatively  high  values  for 
healthcare  under  the  reconstruction  values,  and  relatively  low  military  kinetic  values. 


Figure  4.  Example  of  Unit  Abilities  in  PSOM 
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2. 


Stances 


Stances  provide  a  catalog  of  actions  that  a  unit  or  faction  can  take  toward  another 
entity  in  the  game.  Stances  are  broken  in  categories  based  on  a  main  stance,  and  each 
main  stance  has  one  or  more  sub  stances.  The  stances  are  similar  to  the  operations  and 
the  sub  stances  are  similar  to  tactical  tasks.  For  example,  a  main  stance  can  be  attack  and 
the  sub  stances  are  the  multiple  types  of  attacks,  such  as  ambush  or  indirect  fires.  This 
category  allows  the  user  to  add  main  stances  and  sub  stances  and  make  changes  to  the 
numeric  values  that  create  the  particular  stance  array.  A  unit’s  stance  determines  the 
intensity  of  combat  tasks,  information  operations,  humanitarian  tasks,  or  logistical 
support.  Figure  5  is  a  screen  shot  from  the  stance  “attack”  with  sub  stance  “ambush- 
direct  fire.”  So  a  unit  given  this  stance,  regardless  of  unit  type,  has  the  corresponding 
values.  Also  provided  in  Figure  4  is  a  list  of  the  main  stances  available  to  all  units. 
Special  units  (e.g.,  air  and  sea)  have  specific  stances. 


Data  and  Settings 

Save/Load  Settings  Unit  Abilities  Stances  Goods  Details  Descriptors  Population  Agents  Pre-Set  Unit  Types  Terrain  Display  Colours  Combat  Modifiers  Outputs  Weather 
Main  Stance:  | Attack/Enforce  ~j^| 


Occupy  Infrastructure 
Destroy  Infrastructure 


Indirect  Fire/Harrassment 
Attack  Logs 

Destroy  Human  Capital  (Covert) 
Destroy  Infrastructure  (Covert) 
Ambush  -  IED 
Anti-Air 
Support  Fires 


Name:  Ambush  -  Direct  Fire 


Attack 
Attack  rr — 

Infrastructure: 
Attack  Human  it— 
Capital:  |£_ 


Defend 

Guard  Units:  [o~ 


Reconstruction 


Other  Military  Tasks 
Train  Units:  jo 


Crime  and  Policing 
Traffik:  0 


Other 

Logs  Gj- 
Provision:  — 


Attack  Units: 
Attack  Logs: 


Modify  Perceptions: 
C2  Cost: 


Attack  Population:  0 


on:  0 


kid:  0 


QRF:  1 0.3 


'E 


Average  Size  of  kg 
Force  on  Stance: 


r^r 


Main  Stances 
(Coalition  Units) 

Build  Humanitarian  Aid 

Attack/Enforce 

Control/Stabilize 

Transition 

Information  Operations 
Reconnaissance 
Logistics/Headquarters 
Counter-transition 


Figure  5.  Example  of  Stance  Setting  in  PSOM 
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3. 


Goods  and  Services 


This  category  allows  for  categorical  costs,  production  details,  and  population 
expectations  of  provided  goods  and  services.  Goods  and  services  can  be  provided  by 
participating  factions  as  any  combination  of  palliative  aid,  infrastructure,  and  human 
capital.  Different  types  of  goods  can  be  added  to  the  model,  some  common  examples  of 
which  are  power,  sanitation,  food,  and  security.  The  modeler  can  assign  values  such  as 
cost  per  person  for  palliative  aid,  maintenance  costs,  production  details,  and  the 
population  expectation  for  a  particular  good  or  service.  A  population’s  expectations 
compared  to  what  is  provided  effects  the  population’s  consent  toward  particular  factions. 
The  effect  of  a  particular  good  or  service  provided  in  PSOM  on  population  consent  is 
based  on  public  choice  theory  (Body,  2008,  p.  45). 

4.  Terrain 

PSOM  can  represent  different  types  of  terrain.  Unit  firepower,  protection, 
detection  ability,  and  mobility  are  all  modified  by  terrain  settings. 

5.  Population  Agents 

Agents  are  “Semi  Autonomous  groups  of  ‘similar’  people  of  a  defined  size.  An 
agent  may  make  a  number  of  decisions  based  on  the  conditions  and  environment 
surrounding  it”  (Parkman,  2008,  p.  41).  This  category  is  used  to  create  generalizations 
about  the  overall  population.  The  number  of  people  in  a  particular  agent  (population 
resolution),  group  decision  actions  (decision  radius),  and  criminality  (average  time  in 
prison  for  crime)  settings  are  examples  of  attributes  in  a  population  agent. 

6.  Combat  Modifiers 

These  settings  allow  for  generalizations  about  combative  units.  These  are  often 
averages  that  are  used  in  later  calculations.  Examples  are  average  distance  moved, 
planning  delay,  average  leadership  values,  and  comparative  values  for  conscripts  versus 
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veterans,  and  typical  fire  control  status  (ROE).  These  values  provide  a  sort  of  base  case 
or  expected  values  for  military  or  paramilitary  unit  types  using  subjective  subject  matter 
expert  inputs. 


7.  Outputs 

PSOM  provides  a  wealth  of  deterministic  metrics  that  can  be  displayed  via  the 
graphical  user  interface  (GUI)  or  sent  to  a  comma  separate  value  file  (CSV)  for  analysis. 
Multiple  metrics  relating  to  PMESII  can  be  extracted  from  the  outputs  tab.  This  is 
important  because  in  irregular  warfare  success  is  often  subjective  and  there  is  no  clean 
list  which  provides  the  conditions  for  victory.  Rather,  many  metrics  are  interwoven  and 
dependent  on  one  another.  There  is  more  discussion  on  this  topic  in  the  results  section. 

E.  GAME  PLAY 

PSOM  is  designed  to  be  a  human-in-the-loop  wargame.  By  definition  a  wargame 
is  a  tool  for  the  modeling  and  exploration  of  human  decisions  processes  in  the  content  of 
military  action  (Perla  1990  p.  261). 

An  action  model  that  is  disconnected  from  the  decision  maker’s  intuition 
and  from  the  concepts  he  or  she  is  familiar  with  does  not  permit  interplay 
between  the  decision  maker  and  the  model.  In  short,  complicated,  non- 
intuitive  action  models  require  decision  makers  to  accept  the  implications 
of  the  model  on  blind  faith.  Action  models  should  aid  decision  makers  not 
replace  them.  (Committee  on  Organizational  Modeling:  From  Individuals 
to  Societies,  2008,  p.  321). 

Therefore,  the  way  in  which  the  game  is  played  is  critical.  The  game  should  be 
intuitive  in  how  it  is  played  so  as  to  ensure  the  semantics  of  game  play  do  not  interfere 
with  player  decisions  and  lessons  learned  from  game  play.  PSOM  is  implemented  by  two 
separate  installations,  one  server  and  multiple  clients.  Clients  represent  the  multiple 
factions  and  units  involved  in  the  wargame.  Figure  6  was  taken  from  the  PSOM 
specification  manual  and  displays  this. 
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Figure  6.  Setup  of  Game  Play  [From  (PSOM  Specification  Manual)] 

The  client  interface  shown  in  Figure  7  is  the  primary  means  in  which  the  players 
implement  unit  orders.  Prior  to  each  time  step  the  client  updates  the  client  interface.  The 
client  can  determine  the  unit’s  stance  and  sub-stance  toward  particular  factions,  initiate 
movement,  loosen  or  tighten  the  unit  ROE,  change  the  unit’s  force  protection  level,  along 
with  other  faction-specific  updates.  Once  the  player  (client)  has  updated  this  interface  for 
all  units  within  the  corresponding  faction,  the  player  sends  his  complete  order  to  the 
server.  It  is  possible  for  a  client’s  order  to  be  scripted  as  well.  For  example,  the  game 
director  can  submit  a  predetermined  order  for  a  new  insurgent  group  to  attack  a  target  on 
a  particular  turn. 
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Figure  7.  PSOM  Player  Interface  [Best  viewed  in  color] 


Similar  to  the  client  interface  for  factions,  the  higher  level  game  interface  shown 
in  Figure  8  is  used  to  update  the  political  and  strategic  environment  the  game. 
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Faction  -  Coalition 


rCiient  Mode:  Turn  0  (D  -30) 


Load/Savej  High  Level  Game  Map  Graphs  Intelligence  Headlines  Campaign  Progress 


Strategic  Settings  Region  Map  Balance  Sheet 


Infrastructure/Human  Capital/ Aid  Transfers 
C  None 

O  All  Target  Faction:  | 

Region: 

O  Region  lo 


3 


O  Single  Cell 


*  0 


[  Add  ]  [  Clear  ] 


Wealth  Transfers 


Coalition:  0 

Office  of  the  Martyr  Sadr:  0 
Iraqi  Government:  0 
Sunni  Nationalists:  0 
Civilian:  0 
UN&IGOs:  0 


Legitimacy  Rating 
Rule  of  Law: 


Consent: 
(for  Government) 

Average  Security: 

Employment: 


Attack/Hold  Fire 
O  Attack  O  Hold  Fire 

Intelligence  Sharing 
O  share  O  Don't  Share 


0  Blind 


Save 

Changes 


Cancel 

Changes 


Recruit  New  Units 


Current  Recruits:  0 


Economics 
Tax  Rate: 


:o Load/save  1 1  Save  Changes 


Figure  8.  High  Level  Game  Interface 

Once  all  of  the  clients/players  have  sent  their  updated  orders  to  the  server,  the  next  time 
step  for  the  game  can  be  run  and  the  outcomes  of  the  previous  actions  are  provided.  At 
this  time  the  game  turn  process  is  completed  again. 

Time  steps  for  the  wargame  can  be  variable;  however,  a  normal  time  step  is  thirty 
days.  Therefore,  all  contacts,  humanitarian  actions,  policy  changes,  training  missions, 
population  actions,  and  IW  considerations  for  thirty  days  will  be  played  out  between  each 
turn.  Faction  activities  will  influence  the  population,  resulting  in  changes  in  population 
support,  HUMINT,  crime,  etc.  These  will  all  play  out  during  the  thirty-day  period  and 
have  effects  on  the  status  of  the  host  nation  in  which  the  war  is  being  fought  at  the  end  of 
the  time  step. 
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F. 


OUTPUT 


PSOM  can  output  a  tremendous  amount  of  data  as  output  from  each  turn  of  the 
game.  This  is  imperative  because  “No  definitive  list  can  be  developed  due  to  the 
somewhat  subjective  nature  of  success”  (Body,  2008,  p.  16).  Because  of  the  interaction 
between  factions,  many  measures  of  effectiveness  are  explained  in  regards  to  a 
relationship  between  factions  and  the  population.  This  level  of  detail  allows  particular 
factions  to  know  how  their  conduct  effects  the  population  and  other  factions.  With 
careful  analysis  of  the  multiple  outputs  a  faction  can  determine  the  causality  of  actions  by 
subordinate  units.  However,  these  relationships  are  not  often  apparent  with  the  basic 
outputs  displayed  between  turns  and  take  some  experience  with  PSOM  to  find.  PSOM 
does  not  provide  a  list  of  stochastic  possibilities  that  units  could  take  in  regards  to  faction 
actions. 

In  approximate  accordance  with  military  doctrine,  the  primary  MOEs  for  PSOM 
are  security  and  consent  (Stability  Operations,  2008,  pp.  1-33,  1-77).  Both  of  these 
measures  relate  to  the  population  and  are  represented  on  a  scale  ranging  from  1  to  10.  In 
the  philosophy  documentation  (Body  H.  ,  2008),  security  is  defined  as  the  key  MOE  in 
PSOM.  Security  tracks  the  perceived  risk  of  violent  death  of  a  population  agent  in  a 
scenario.  This  MOE  is  applied  only  to  the  population  agents  in  regard  to  factions.  It  is  an 
aggregate  of  the  security  amongst  all  ethnic  groups  within  that  population’s  grid  square. 
The  security  MOE  is  subject  to  a  memory  effect  in  that  perception  takes  time  to  catch  up 
with  reality.  It  is  calculated  from  deaths  to  an  ethnic  group  within  a  cell  while  accounting 
for  the  size  of  that  ethnic  group.  If  a  faction  actively  provides  security  to  a  population  the 
metric  will  credit  the  faction  with  the  security  provided. 

The  consent  MOE  depicts  the  degree  to  which  the  population  supports  and 
submits  to  the  will  of  a  faction  (Jon  Parkman,  2008).  Unlike  the  security  MOE,  consent 
is  not  an  aggregate  of  ethnic  groups;  rather,  it  is  based  on  the  perception  of  a  particular 
ethnic  group  toward  a  faction.  So,  for  example,  one  ethnic  group  can  have  a  high  consent 
toward  the  national  government  while  another  ethnicity  is  the  opposite.  The  consent 
MOE  is  subject  to  memory  effects  and  it  is  modified  by  the  ideological  differences 
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represented  by  the  Nolan  Chart.  The  value  of  consent  in  regard  to  each  faction  is  a 
function  of  the  level  of  goods  and  services  provided,  the  marginal  gain  coefficients,  and 
the  population  agents’  expectations.  This  approach  is  supported  by  the  standard  rational 
choice  theory.  “The  basic  idea  behind  rational  choice  theory  is  that  people  do  their  best 
under  prevailing  circumstances”  (Green,  2002,  p.  5).  Although  these  are  the  primary 
outputs,  Table  2  shows  the  PMESII  indicators  and  their  equivalent  PSOM  output. 


PMESII  Indicator 

Corresponding  PSOM  Output 

Political  Legitimacy 

Population  Consent  toward  own  Government,  Security, 
Rule  of  Law,  Corruption,  Provision  of  Essential  Services 

Military 

Casualties,  Contacts 

Economic 

Production,  Reconstruction,  Income,  Human  Capital, 
Growth  Rate 

Social 

Fear,  Rule  of  Law,  Economic  Factors 

Information 

HUMINT,  Headlines 

Infrastructure 

Infrastructure,  Economic  Production,  Human  Capital 

Table  2.  PMESII  Indicators  in  Relation  to  PSOM  Output 


PSOM  also  provides  the  user  a  headlines  function  which  is  intended  to  highlight 
activities  that  would  most  likely  become  open  source  press  releases.  Such  topics  as 
civilian  deaths,  enemy  deaths,  and  terrorist  attacks  are  general  headlines. 

The  taxonomy  of  the  parameters  and  the  responses  for  the  Peace  Support 
Operations  Model  provides  a  logical  and  intuitive  environment  to  simulate  contemporary 
warfare.  As  explained  earlier  in  this  chapter,  the  inputs  required  to  design  and  execute  a 
simulation  in  PSOM  cover  a  vast  space  from  combat  capabilities  to  social  behaviors. 
This  study  is  able  to  take  advantage  of  this  design  to  create  an  experiment  that  will 
evaluate  the  model  based  on  its  underlying  categorical  blueprint.  By  initially  completing 
multiple  designs  of  experiments  based  on  the  model’s  categorical  setup,  we  can  then 
determine  and  aggregate  the  influential  factors  to  further  explore  the  model  in  its  entirety. 
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III.  SCENARIO  DESIGN 


One  of  the  most  important  jobs  of  wargame  developers  is  to  assess  the 
validity  of  the  game’s  results  and  processes  in  light  of  the  real  world. 

(Perla,  1990) 


A.  INTRODUCTION 

In  evaluating  military  simulation  models  it  is  critical  to  use  a  realistic  and 
accurate  scenario  that  is  pertinent  to  current  military  operations.  This  chapter  describes 
the  macro  level  military,  political,  and  societal  scenario  used  for  all  experimentation.  The 
chapter  then  describes  the  focus  areas  of  the  experimentation  with  more  resolution.  This 
includes  a  description  of  factions,  coalition  military  units,  and  ethnic  groups  that  are 
critical  in  the  analysis.  The  chapter  concludes  with  a  summary  of  the  measures  of 
effectiveness,  units  employed,  and  stances  in  use. 

B.  SCENARIO  CHOICE 

1.  Overview 

Because  of  the  considerable  complexity  of  a  campaign  level  wargame  that 
portrays  the  importance  of  the  population,  the  first  priority  is  using  a  scenario  that  is 
based  on  thorough  research  and  factual  data.  Secondly,  the  scenario  should  describe  a 
military  situation  that  is  relatively  well  understood,  thus  allowing  intuitive  and  doctrinal 
analysis.  Therefore,  we  used  a  scenario  developed  by  DSTL  based  on  Iraq  in  the  time 
frame  of  2004  onward.  DSTL  developed  this  scenario  to  test  PSOM  and  has  conducted 
multiple  wargames  and  considerable  adjustments  to  ensure  scenario  accuracy.  Also, 
PSOM  is  a  fairly  new  model  with  minimal  documentation  and  thousands  of  possible 
inputs;  therefore,  for  this  study,  we  felt  it  best  to  use  a  scenario  designed  by  developers 
who  actually  understand  the  simulation. 
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Although  the  scenario  encompasses  all  of  Iraq  in  the  given  time  frame,  in  order  to 
allow  focused  quantitative  analysis  this  study  primarily  concentrates  on  the  coalition 
forces,  the  Sunni  Nationalists,  and  the  Sunni  Ethnic  Group  Population  Agents.  All 
factions  and  ethnic  groups  are  discussed  in  this  section;  however,  the  emphasis  is  on  the 
aforementioned  groups. 

2.  The  Macro  Level  Scenario 

As  previously  stated,  the  scenario  used  is  Iraq  2004  onward.  The  force  structure 
of  coalition  forces  closely  matches  that  of  mid  2004  to  2005.  The  enemy  force  structure 
is  loosely  based  on  known  information,  and  the  population’s  attributes  are  also  reflective 
of  this  geo-spatial  information.  Table  3  describes  the  factions  in  play  and  their  respective 
sub-factions.  For  the  majority  of  simulation  runs,  twelve  30-day  time  steps,  equivalent  to 


one  year  of  combat,  are  played.  Therefore,  the  game  is  a  terminating  scenario. 


Coalition 

Office  of  the  Matyr 
Sadr 

Iraqi  Governemnt 

Sunni  Nationalists 

UN,  IGO’s 

Combined  Air 
Operations  Center 

Jaish  al-Mahdi 
Commander 

New  Iraqi  Army 

Ba’athist 

United  Nations 
Assistance  Mission 
for  Iraq 

Combined  Joint 
Special  Operations 
Task  Force 

Police  Service 

Al-Qaeda  in  Iraq 

Multi  National 
Division  (B) 

Facility  Protection 
Service 

Tribal  Militias 

Multi  Nation 

Division  (NC) 

Development  Center 

Multi  Nation 

Division  (CS) 

Shi’ a  Militias 

Multi  National 
Division  (SE) 

National  Police 

Multi  National 
Division  (W) 

Coalition 

Provisional 

Authority 

Multi  National 

Forces  Iraq  Corps 
Assets 

Table  3.  Breakdown  of  Iraq  Scenario  Factions 


26 


The  population  consists  of  three  primary  ethnic  groups  (using  the  PSOM 
definition  of  ethnic  group):  Sunni,  Shi’a,  and  Kurds.  During  Saddam  Hussein’s  reign 
these  ethnic  groups  separated  themselves  across  the  country,  creating  geo-ethnic  areas 
throughout  the  entire  nation.  For  this  particular  scenario  the  coalition  defined  areas  of 
responsibility  closely  align  with  these  ethnic  divides.  This  is  shown  below  in  Figure  9. 
Due  to  the  sheer  size  and  complexity  of  this  scenario,  we  have  focused  our  emphasis  on 
Sunni  civilians,  the  Sunni  Nationalist  factions,  and  the  coalition  forces  that  are  assigned 
to  the  areas  in  which  Sunni’s  and  the  Sunni  Nationalists  live  and  operate.  Figure  9 
displays  the  Office  of  the  Martyr  Sadre  Faction  (JAM),  the  Sunni  Nationalist  Faction,  and 
the  population  of  ethnic  groups  by  region  at  simulation  time  step  zero. 
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Figure  9.  Break  Down  of  Population,  JAM,  and  Sunni  Nationalists  by  AOR  [Best 

viewed  in  color] 
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Figure  10  shows  the  Map  of  Iraq  broken  down  by  Area  of  Responsibility  used  for 
this  study. 
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Figure  10.  Coalition  Force  Structure  by  Area  of  Responsibility  [Best  viewed  in  color] 


C.  SITUATION 

By  late  2004,  coalition  forces  were  well  established  with  over  150,000  troops  in 
Iraq.  Although  the  new  Iraqi  government  has  control  over  the  country,  the  population’s 
support  is  questionable  across  the  nation.  The  growth  of  A1  Qaeda  in  Iraq  (AQI)  has 
been  noted  throughout  the  country  and  increased  violence  between  ethnic  groups  has 
many  feeling  a  civil  war  is  inevitable.  At  the  same  time,  a  tremendous  emphasis  has  been 
put  forth  by  the  coalition  to  build  Iraqi  Security  Forces  and  the  Iraqi  Army  is  starting  to 
take  shape.  The  country’s  infrastructure  is  still  weak,  proven  by  a  lack  of  continuous 

power  and  potable  water  throughout  the  country. 
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1. 


Factions 


a.  Coalition  Forces 

The  faction  of  coalition  forces  consists  of  over  sixty  combat  maneuver 
battalions  and  the  required  combat  support  and  service  support.  These  forces  are  spread 
throughout  the  country  with  the  majority  focused  on  Baghdad  and  MND(NC).  In 
addition  to  the  combat  units,  numerous  air  wings,  indigenous  force  training  teams,  and 
reconstruction  teams  support  the  coalition  mission.  The  coalition’s  primary  stances 
during  this  phase  of  the  war  are  to  provide  security  and  assist  in  the  rebuilding  of  the  war- 
tom  nation. 


b.  Sunni  Nationalists 

The  faction  of  Sunni  Nationalists  consists  of  the  emersion  of  Al-Qaida  in 
Iraq  (AQI)  and  tribal  militias.  AQI  consists  of  ten  cells  each  with  approximately  forty 
members.  Their  primary  focus  is  either  attacking  the  coalition  forces  or  the  Iraqi 
government  through  the  destruction  of  infrastructure.  As  can  be  seen  in  Figure  9,  the 
majority  of  these  forces  are  in  Bagdad  or  the  surrounding  areas.  The  majority  of  the 
Sunni  Nationalists  are  the  nine  tribal  militias  which  consist  of  approximately  700  troops 
each.  These  units  are  primarily  anti-coalition  and  anti-Iraqi  government  as  well.  The 
Sunni  Nationalists  also  maintain  a  small  number  of  training  and  reconstruction  teams 
which  operate  in  the  central  to  northern  portions  of  the  country. 

c.  Office  of  the  Martyr  Sadr 

The  faction  of  the  Office  of  the  Martyr  Sadr  (JAM)  consists  of  fourteen 
militia  battalions,  but  for  purposes  of  this  scenario  only  seven  are  activated  and  seven  are 
in  hiding.  In  addition,  JAM  has  seven  training  units  and  seven  construction  units  to  help 
win  populace  consent.  As  can  be  seen  in  Figure  8,  the  only  area  where  JAM  and  Sunni 
Nationalists  both  operate  is  in  Baghdad;  the  remaining  JAM  units  focus  primarily  in 
MND(CS)  and  MNS(SE).  JAM’s  primary  stance  in  this  game  is  to  protect  the  Shi’a 
population  from  both  the  Sunni  Nationalists  and  the  coalition  forces. 
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d.  Iraqi  Government 


The  faction  representing  the  Iraqi  government  primarily  consists  of 
security  forces.  The  Iraqi  Security  Forces  are  comprised  of  forty  police  Battalions,  forty 
facility  protection  service  (FPS)  groups,  five  construction  teams,  five  training  teams, 
twelve  Militia  Battalions,  twenty-seven  national  police  battalions,  and  some  additional 
recruiting  and  information  operations  units.  These  units  are  primarily  focused  on  the 
security  of  the  nation. 


e.  United  Nations  and  Inter-governmental  Organizations  (UN, 

IGO) 

The  UN  and  IGOs  faction  consists  of  thirty-three  humanitarian  aid  teams 
spread  throughout  the  nation.  Their  primary  role  is  to  provide  aid  in  the  form  of  medical 
assistance,  education,  subsistence,  and  infrastructure  to  the  population. 

2.  Population 

a.  Sunni  Ethnic  Group 

In  this  scenario  the  Sunni  population  comprises  approximately  25%  of 
Iraq’s  total  population  (see  Figure  8).  As  a  group,  the  Sunni  do  not  approve  of  the 
coalition  forces  and  have  only  a  slightly  higher  opinion  of  the  current  Iraqi  government 
than  of  the  coalition  forces.  However,  the  Sunni  population  has  very  strong  consent  for 
the  Sunni  Nationalists  and  their  cause.  The  Sunni  population  holds  the  Office  of  the 
Martyr  Sadr  in  contempt  and  fears  JAM  and  coalition  forces  equally.  The  population 
primarily  lives  in  the  north  central  areas  of  Iraq  and  except  for  in  Baghdad  is  segregated 
from  the  Shi’ a  populace. 
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b. 


Shi’ a  Ethnic  Group 


In  the  given  scenario  the  Iraqi  population  is  48%  Shi ’a.  As  a  group,  the 
Shi’ a  accept  coalition  forces  with  consent  equivalent  to  the  Office  of  Martyr  Sadr.  The 
Shi’ a  generally  support  the  current  Iraqi  government,  but  condemn  the  Sunni 
Nationalists.  The  Shi’a  primarily  reside  in  Baghdad  and  southern  Iraq. 

c.  Kurd  Ethnic  Group 

The  Kurd  ethnic  group  provides  for  the  remaining  25%  of  the  Iraqi 
population.  They  primarily  live  in  the  most  northern  areas  of  Iraq  and  support  both  the 
coalition  forces  and  the  Iraqi  government.  The  Kurds  are  impartial  toward  the  Office  of 
the  Martyr  Sadr,  but  hold  the  Sunni  Nationalists  in  contempt. 

3.  Game  Execution 

a.  Overview 

Normally,  the  intent  of  the  coalition  faction  in  this  wargame  is  to  actually 
play  the  PSOM  Iraq  scenario  with  the  goal  of  creating  an  independent,  secure  Iraq  with 
lucrative  economic  conditions  and  a  legitimate  government  fully  supported  by  the 
population.  However,  this  is  not  the  case  for  this  experiment.  Rather,  our  goal  is  to  test 
the  realm  of  possible  outcomes  received  by  the  users  of  PSOM.  For  this  we  focus  on  the 
security  levels  and  consent  levels  throughout  the  country  as  we  change  faction  activities 
and  scenario  parameters. 

The  majority  of  simulations  executed  represent  one  year  of  combat. 

Starting  conditions  were  not  changed  within  simulation  runs,  so  if  a  faction  begins  the 

year  with  an  aggressive  stance  and  a  relaxed  set  of  ROE,  these  characteristics  are 

maintained  for  the  entire  year.  This  is  an  identified  limit  of  the  study.  The  human-in-the- 

loop  technique  allows  for  factions  to  adapt  to  reactions  of  the  population  and  other 

factions.  Unfortunately,  in  using  the  pre-scripted  batch  technique  to  run  the  scenario  we 

cannot  adjust  the  factions’  actions  throughout  the  year.  This  also  pertains  to  unit 

movement.  Therefore,  a  unit  could  not  move  out  of  its  initial  allotted  footprint.  This 
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being  said,  because  each  time  step  represents  thirty  days  of  combat,  results  of  the 
previous  month  are  carried  over  to  the  current  month’s  fight.  For  example,  civilian  and 
military  causalities,  population  reaction,  and  infrastructure  growth  all  are  determined  time 
step  by  time  step  throughout  the  game. 

b.  Measures  of  Effectiveness 

The  primary  measures  of  effectiveness  addressed  are  the  consent  of  the 
Sunni  population  toward  the  coalition  forces,  the  consent  of  the  Sunni  population  toward 
the  Iraqi  government,  and  the  security  level  throughout  the  country.  PSOM  displays 
these  outputs  in  50km  X  50km  grid  squares  (see  Figure  11).  This  system  results  in  135 
separate  responses  as  the  game  is  played.  To  allow  for  analysis,  the  responses  are 
aggregated  by  coalition  Area  of  Responsibility  and  the  entire  country.  This  allows  for 
regional  averages  for  the  MOEs,  and  therefore  reduces  135  outputs  for  three  metrics  to 
seven  outputs  for  the  three  metrics.  However,  these  metrics  are  based  on  geo-spatial 
representation  and  not  the  quantity  of  people  within  the  cell. 
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Figure  11.  The  Graphical  Display  of  Ethnic  Groups  Consent  for  the  Coalition  from 
the  Peace  Support  Operations  Model  [Best  viewed  in  color] 
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c. 


Units  Played 


Due  to  the  nature  of  a  campaign  level  model,  the  number  of  maneuverable 
units  can  overwhelm  analysis,  causing  excessive  noise  and  confounding.  To  prevent  this, 
the  study  focuses  on  varying  the  attributes  and  actions  of  selected  units  and  population 
agents.  A  total  of  thirty-seven  coalition  maneuver  battalions  which  are  located  in 
MND(B),  MND(NC),  and  MND(W)  are  played,  seventy-five  Sunni  Nationalist  units  are 
played,  forty-seven  UN  IGOs  are  played,  and  535  Sunni  population  agents,  which 
represent  5.35  million  Sunni  civilians,  are  the  population  focus. 

d.  Unit  Stances 

The  complete  taxonomy  of  stances  and  sub  stances  in  this  scenario  ranges 
the  possible  set  of  doctrinally  defined  operational  and  tactical  tasks.  For  this  study, 
coalition  forces  are  limited  to  the  following  stances:  providing  aid,  securing,  attacking, 
and  withdrawn.  Sunni  Nationalists  are  either  attacking  coalition  forces,  attacking  Iraqi 
infrastructure,  or  withdrawn.  IGO’s  are  either  providing  aid  or  withdrawn.  The  stances 
are  uniform  across  the  units  mentioned  in  the  previous  paragraph.  So,  if  a  particular 
design  point  includes  coalition  stances  with  a  stance  of  “withdrawn,”  all  thirty-seven 
coalition  maneuver  battalions  are  withdrawn.  This  is  a  noted  limitation  to  this  particular 
study  and  should  be  examined  further  in  a  follow-on  study. 

D.  SUMMARY 

The  contemporary  Iraq  scenario  used  for  this  study  was  developed  and  thoroughly 
evaluated  and  updated  by  the  Defense  Science  Technology  Lab  UK,  who  also  developed 
the  Peace  Support  Operation  Model.  It  is  the  most  robust  scenario  developed  for  PSOM 
and  the  best  for  this  study.  This  is  of  significant  importance  due  to  the  tremendous 
amount  of  social  parameters  that  a  PMESII  model  must  account  for.  Due  to  the  number 
of  parameters  and  units  the  study  narrows  the  scope  of  concern  to  primarily  the  Sunni 
population,  the  Sunni  Nationalist  Faction,  and  the  coalition  maneuver  battalions  operating 
in  the  areas  where  the  Sunni  population  exists.  By  narrowing  this  area  of  focus  the  study 
can  use  intelligent  Design  of  Experiment  in  conjunction  with  modem  computing  power 

and  basic  data  mining  to  analyze  the  PSOM’s  output. 
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IV.  EXPERIMENTAL  DESIGN 


Everything  should  be  made  as  simple  as  possible,  but  not  simpler. 

Albert  Einstein  1933 


A.  INTRODUCTION 

In  the  simplest  form  the  purpose  of  this  research  is  to  gain  a  basic  understanding 
of  the  Peace  Support  Operation  Model.  In  particular,  we  need  to  determine  which  factors 
are  significant  in  the  model  and  how  they  affect  the  responses  (Kelton,  1991,  p.  657).  In 
order  to  quantifiably  analyze  PSOM,  this  study  follows  the  data  farming  process 
described  by  Home  (2004).  Our  implemented  iterative  data  farming  process  entails  a 
four  step  procedure:  Define  factors  of  interest,  create  a  design  of  experiment  (DOE),  run 
the  simulation  experiments  in  parallel  on  a  computing  cluster,  and  conduct  data  mining. 
The  initial  definition  of  interesting  factors  is  a  research  and  subject  matter  centric  step  to 
begin  the  process.  The  use  of  DOE  allows  us  to  gain  “detailed  insight  into  the  model’s 
behavior”  (Kleijnen,  2005,  p.  266),  control  the  bias  and  confounding  of  factors,  and 
explore  a  tremendous  number  of  possible  parameter  combinations  efficiently.  Data 
mining  is  then  used  to  analyze  responses  from  the  created  vast  parameter  space  (Phillip 
Barry  &  Koehler,  2004).  This  is  essential  as  a  low-resolution  model  which  encompasses 
social  implications  has  no  closed  form  solution.  We  can  then  take  this  large  data  set  and 
develop  relatively  simple  formulas  that  act  as  a  proxy  for  the  actual  simulation  (Kelton, 
1991).  These  “meta  models”  can  then  be  used  to  determine  which  factors  are  significant 
within  PSOM  and  what  effects  their  manipulation  might  have. 

This  chapter  describes  in  detail  the  variables  selected  as  factors  in  the  experiment, 
the  methodology  used  to  create  the  DOEs,  and  the  tools  created  at  the  Naval  Post¬ 
graduate  School  to  take  the  PSOM  wargame  and  convert  it  into  a  data  farmable 
simulation.  Next,  the  design  of  experiments  processes  used  are  discussed,  providing  the 
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reader  insight  into  how  the  study  efficiently  tests  a  large  space  of  possible  combinations 
of  input  parameters  (Home,  2004).  We  then  discuss  the  meta  modeling  techniques  used 
to  data  mine  the  tremendous  amount  of  output  collected  from  the  experiments. 

B.  FACTOR  SELECTION  AND  METHODOLOGY 

1.  Overview 

In  analyzing  the  variables  and  comparing  results,  factor  selection  proves  vital  to 
an  efficient  and  worthwhile  study.  PSOM  is  a  complex  model  and  the  number  of  factors 
used  to  build  the  Iraq  scenario  is  too  immense  to  study  in  its  entirety.  For  example,  there 
are  twelve  factors  used  to  describe  each  particular  unit  and  there  are  an  additional  thirty- 
three  attributes  used  to  create  each  unit  type.  For  example,  A/1-26  IN  has  twelve  specific 
settings  and  every  generic  infantry  company  has  thirty-three  attributes  in  common  with 
all  infantry  companies.  Therefore,  any  one  of  the  200  coalition  units  has  forty-five 
variables  which  describe  its  totality.  In  addition  to  the  number  of  factors,  there  is  no 
published  PSOM  user’s  manual  (there  is  a  draft  manual  cited  throughout  this  work),  and 
many  of  the  variables  are  not  constrained  to  particular  numeric  ranges.  Naively  selecting 
the  factors  without  regard  to  their  appropriate  ranges  would  result  in  a  design  where 
important  factors  are  washed  out  by  noise  or  dominated  by  unknown  limits  on 
parameters.  This  section  explains  the  tactics  used  and  factors  selected  in  building  the 
experiment. 

As  previously  stated,  the  scope  of  the  study  is  limited  by  the  focus  on 
geographical  areas  inhabited  by  the  Sunni.  To  further  narrow  the  scope,  we  looked  to  the 
opinion  of  subject  matter  experts  on  irregular  warfare,  combat  modeling,  PSOM,  and  the 
DoD’s  needs  in  IW  modeling  for  the  first  stages  of  the  experimental  design.  Although 
this  reduces  the  number  of  factors  tremendously,  multiple  designs  were  needed  to  create 
practical  experiments.  These  experiments  are  categorized  by  scenario  design  and  settings 
design.  Finally,  because  this  is  an  iterative  process,  the  emersion  of  results,  analysis,  and 
expert  opinion  creates  a  cumulative  design  of  experiment. 
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2. 


Scenario  Specific  Design  of  Experiment 


The  scenario  specific  design  of  experiment  analyzes  the  parameters  found  in  the 
scenario  file.  These  factors  are  either  set  during  scenario  development  or  changed  by 
players  during  the  game.  Many  of  these  factors  are  crucial  assumptions  made  about 
either  population  agents  or  factions.  Of  note,  not  all  factors  are  quantitative  in  nature. 
For  example,  a  unit’s  stance  is  an  action  from  a  list  of  possible  tactical  or  operational 
tasks.  These  categorical  factors  increase  the  number  of  design  points  tremendously  and 
are  therefore  limited  in  scope.  Table  4  shows  the  list  of  scenario  design  categorical 
factors,  their  experimental  ranges,  and  brief  descriptions.  Stance  changes  affect  all  units 
in  the  area  of  game  play  at  the  beginning  of  the  game  and  they  maintain  these  stances  for 
the  entire  12-month  period. 


Factor 

Level  1 

Level  2 

Level  3 

Level  4 

Description 

Coalition 

Stance 

Combat  Units 

CBT  Units 

CBT  Units 

CBT  Units 

37  Coalition 

Attacking 

Securing 

Providing 

Withdrawn 

maneuver 

Sunni 

Humanitarian  Aid 

BN’s  in  Sunni 

Nationalists- 

inhabited 

Sub  Stance 

Build 

regions  take 

Clear 

Patrol 

Infrastructure 

this  stance 

Sunni 

Attacking  U.S. 

Attacking  Iraqi 
Government 

Withdrawn 

75  Sunni 

Nationalist 

Nationalist 

Stance 

Destroy 

Infrastructure 

Units  (AQI 
and  Militia) 
in  Sunni 
inhabited 

Sub  Stance 

Ambush 

regions  take 
this  stance 

IGO  Stance 

Withdrawn 

Provide 

47  IGOs 

Humanitarian 

Aid 

throughout 

Iraq  take  this 

stance 

Coalition 

Yes 

No 

Determines  if 

Shares 

the  coalition 

Intelligence 

and  Sunni 

with  Sunni 

Nationalist 

Nationalist 

share 

information 

Table  4.  Categorical  Factors  used  in  the  Scenario  DOE 
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Table  5  is  a  list  of  all  the  quantitative  factors  in  the  scenario  DOE,  their 
experimental  range,  and  a  short  description. 


Factor 

Experimental 

Range 

Description  (Parkman,  2008) 

Coalition 
ROE  Level 

1-5 

1  (Loose)  and  5  (Tight)  representing  the  degree  to  which  the  unit  is  willing  to 
cause  civilian  casualties  in  order  to  complete  its  task. 

Coalition 

Force 

Protection 

Level 

1-5 

An  integer  between  1  (Low)  and  5  (High)  representing  the  degree  to  which  the 
unit  is  willing  to  suffer  its  own  casualties  in  order  to  complete  its  tasks. 

Sunni  ROE 
Level 

1-5 

See  above. 

Sunni  Force 

Protection 

Level 

1-5 

See  above. 

Sunni 

Political 

ideology 

0-100 

This  is  a  value  between  0  and  100,  which  give  the  Faction’s  ideology,  based 
on  its  views  on  personal  freedom,  through  the  Nolan  chart  system,  as  shown 
below. 

Sunni 

Marginal 

Gains 

0.3— 0.6 

These  values,  one  for  each  Good  Type,  control  the  level  of  importance  that  the 
group  places  on  the  provision  of  that  good  type. 

Sunni 

Marginal 

Gains 

Security 

0.3— 0.6 

This  value  controls  the  level  of  importance  that  the  ethnic  group  places  on 
Security. 

Sunni  Initial 

Consent 

Coalition 

2-8 

These  values  set  the  initial  levels  of  Consent  towards  each  Faction  that  are 
possessed  by  the  generated  Agents  at  the  start  of  play. 

Sunni  Initial 
Threat 
toward 
coalition 

2-8 

This  value  sets  the  initial  level  of  threat  that  a  population  agent  feels  toward 
each  faction. 

Coalition 

Casualty 

tolerance 

0-100 

Casualty  tolerance  value,  which  controls  how  many  casualties  the  unit  will 
bear  each  turn  before  the  deterrence  function  begins  to  have  an  effect. 

Coalition 

Leadership 

0-100 

The  level  of  competence  in  the  leadership  of  the  unit. 

Coalition 

Experience 

0-100 

The  level  to  which  the  Unit  is  trained  and  experienced  in  conducting 
operations  in  a  PSO  type  situation. 

Coalition 

Reputation 

0-100 

The  degree  to  which  the  population  perceives  that  the  unit  is  unwilling  to 
conduct  offensive  operations  against  them. 

Coalition 
Turns  at 
Location 

0-12 

The  values  for  calculating  the  unit’s  familiarity  with  the  local  environment. 

Table  5.  Continuous  Factors  used  in  the  Scenario  DOE  [From  (Jon  Parkman,  2008)] 
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3.  Explanation  of  Factors 


a.  Stance 

The  stance  that  a  faction  takes  determines  the  actions  of  that  faction. 
Therefore,  the  choice  of  stance  is  critical.  This  experimental  design  emphasizes  the 
different  initial  stances  and  the  resulting  responses.  If  the  players’  actions  do  not  provide 
reasonable  reactions  from  the  game,  then  its  use  is  questionable.  We  expected  to  see 
considerable  interaction  between  the  coalition  and  Sunni  Nationalist  stances.  This  thesis 
looks  into  the  stances  of  the  coalition  forces,  Sunni  Nationalists,  and  the  IGOs.  These 
factors  are  normally  controlled  by  the  players. 

b.  Rules  of  Engagement 

The  players  also  set  each  unit’s  ROE.  This  provides  the  player  the  ability 
to  designate  the  degree  in  which  each  unit  is  willing  to  cause  civilian  casualties  in  order 
to  accomplish  its  mission.  A  setting  of  1  is  a  very  loose  ROE  and  would  be  indicative  of 
a  VBIED  detonated  in  a  civilian  populated  area,  where  as  an  ROE  of  5  represents  a  sniper 
who  would  only  engage  upon  positive  identification  of  an  enemy. 

c.  Force  Protection 

Force  protection  is  similar  to  ROE  in  that  the  players  control  this  setting 
for  each  unit  on  a  scale  from  1-5.  A  unit’s  force  protection  describes  its  willingness  to 
assume  risk  to  accomplish  its  mission.  A  value  of  1  is  low  risk  and  a  value  of  5  is  high 
risk. 


d.  Political  Ideology 

Political  ideology  is  a  population  agent  attribute  which  falls  on  a  scale  of 
1-100  (see  Chapter  II).  This  ideology  is  important  in  the  development  of  population 
agents  and  therefore  should  play  a  role  in  the  population’s  consent  toward  factions. 
Ideologies  are  defined  in  the  scenario  setup. 
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e.  Marginal  Gains  for  Goods 

Marginal  gains  determine  the  importance  of  a  particular  good  for  each 
ethnic  group.  Marginal  gains  are  on  a  scale  of  0-1  and  are  a  primary  characteristic  in  the 
building  of  population  agents  during  scenario  development.  Because  these  play  a  key 
role  in  the  population’s  development  and  rules  it  is  essential  to  understand  their  impact  on 
the  game.  During  the  initial  iterations  this  factor  proved  very  powerful  to  game 
outcomes,  so  the  experimental  range  was  limited  to  0.3-0. 6. 

f.  Marginal  Gains  for  Security 

The  security  marginal  game  is  similar  to  that  for  goods.  In  this  study  its 
value  was  changed  separately  because  a  society’s  concern  for  security  is  more  likely  to  be 
influenced  based  on  combat  actions.  Thus,  the  consent  corresponding  to  marginal  gain 
security  is  changed  based  on  security  provided  as  opposed  to  goods  production.  During 
the  initial  iterations  this  factor  proved  very  powerful  to  the  game  outcome  so  the 
experimental  range  was  limited  to  0.3-0. 6. 

g.  Initial  Consent 

The  initial  consent  an  ethnic  group  feels  toward  a  faction  is  a  tremendous 
assumption  in  scenario  development.  As  with  consent,  this  factor  is  on  a  scale  from  0  to 
10.  With  any  scenario  it  is  crucial  to  understand  how  initial  conditions  can  affect 
simulation  execution. 

h.  Initial  Threat 

The  initial  assumption  of  threat  an  ethnic  group  feels  toward  a  faction  is 
similar  to  initial  consent,  and  its  implications  should  be  explored. 

i.  Coalition  Casualty  Tolerance 

Coalition  casualty  tolerance  controls  how  many  casualties  the  unit  will 
bear  each  turn  before  the  deterrence  function  begins  to  have  an  effect.  This  value  is  on  a 
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scale  of  1  to  100  and  is  used  to  describe  each  unit  during  scenario  development.  Often  in 
irregular  warfare  numerically  small  levels  of  casualties  can  create  strategic  implications. 

j.  Coalition  Leadership 

Coalition  leadership  describes  the  level  of  competence  in  the  leadership  of 
a  particular  unit.  This  value  is  on  a  scale  of  1  to  100  and  is  used  to  describe  each  unit 
during  scenario  development.  The  United  States  relies  heavily  on  leadership,  making  this 
factor  an  essential  assumption. 

k.  Coalition  Reputation 

Coalition  reputation  describes  the  perception  the  population  has  on  a 
particular  unit.  This  value  is  on  a  scale  of  1  to  100  and  is  initialized  during  scenario 
development.  In  warfare,  dealing  with  the  populace  the  reputation  of  a  unit  should  have  a 
direct  effect  on  the  collection  of  HUMINT,  coalition  freedom  of  movement,  and  trust. 

l.  Turns  at  Location 

Coalition  turns  at  location  describes  the  number  of  time  steps  which  a  unit 
has  been  in  a  particular  location  and  thus  how  familiar  it  is  with  that  area.  This  value  is 
on  a  scale  of  0  to  100  and  is  initialized  during  scenario  development.  This  factor  has 
implications  in  the  strategic  emplacement  of  units  and  deployment  timelines. 

4.  Setting  Specific  Design  of  Experiment 

The  settings  design  of  experiments  focuses  completely  on  the  settings  factors  in 
the  Peace  Support  Operation  Model.  These  factors  are  primarily  generic  in  nature  in  that 
they  are  descriptions  of  units,  populations,  and  conditions  by  type  rather  than  tied  to  the 
particular  scenario.  The  settings  describe  broad  unit  abilities,  dictate  stance  attributes, 
and  create  values  used  to  modify  mathematical  functions  within  the  simulation.  For 
example,  these  parameters  can  be  used  to  provide  insight  on  increasing  unit  manpower  or 
mobility,  giving  combat  units  the  ability  to  provide  humanitarian  aid,  make  combat  more 
aggressive  in  nature,  or  determine  if  a  patrolling  unit  acts  to  protect  civilians  or  gain 
intelligence. 
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Because  of  the  large  number  of  these  factors  (sixty-six)  and  their  similar  nature,  it 
is  best  to  describe  the  purpose  of  their  exploration  upfront  and  give  only  a  brief 
description  of  each.  First  off,  it  is  crucial  to  understand  the  parameters  used  in  any 
model  prior  to  using  it  as  a  decision  aid  or  training  tool.  Many  of  these  parameters  are 
not  well  defined  in  any  literature,  nor  do  they  have  limits  to  their  values.  Using  an 
intelligent  DOE  we  can  explore  the  space  of  these  factors  and  determine  which,  if  any, 
will  have  a  profound  effect  on  the  simulations  outcome.  This  can  only  lead  to  better  use 
of  the  model  as  a  training  wargame  and  analytic  tool.  Secondly,  many  of  these  attributes 
are  quantitative  descriptions  of  unit  types.  Often  military  comparative  studies  are  used  to 
modify  unit  attributes  in  determining  fielding  and  acquisition  questions.  If  PSOM  can 
provide  such  comparative  analysis  in  the  world  of  irregular  warfare  it  can  prove  a  useful 
simulation.  In  order  to  mitigate  any  one  factor  from  dominating  the  experiment,  these 
factors  are  varied  ±20%  from  the  original  values  in  the  scenario  developed  by  DSTL. 
Table  6  lists  all  the  factors  used  in  the  settings  DOE. 


Factor 

Experimental 

Range 

Description 

Unit  abilities  set  attributes  of  the  particular  unit  types.  These  were  cast  across  maneuver  companies. 

Unit  Fire  Power 

64-96 

Level  of  firepower  per  man 

Unit  Protection 

80-120 

Level  of  armor  per  man 

Unit  Sensor 

40-60 

Level  of  sensors  per  man 

Unit  Intelligence 

16-24 

The  ability  per  man  to  recognise  and  classify 

Unit  Physical  Camouflage 

0.8-1. 2 

Level  to  which  the  element  can  blend  in 
surroundings 

Unit  Social  Camouflage 

0.8-1. 2 

As  above  but  in  social  population 

Unit  Mobility 

24-36 

Average  speed  in  km/h  the  element  moves 

Unit  Change  Attitude  Ability 

2.4-3. 6 

Ability  to  change  the  attitude  of  the  population 

Unit  Crime 

1. 6-2.4 

Ability  per  man  in  the  element  to  commit  crimes 

Unit  Policing 

2.4-3. 6 

The  ability  per  man  in  the  element  to  counter  crime 

Unit  Collateral  Damage 

5.6— 8.4 

The  level  of  expected  collateral  damage  by  unit 

Unit  Size 

72-108 

Number  of  men  in  the  unit 

Unit  Palliative  Aid  Ability 

0.24-0.36 

Unit’s  ability  to  provide  palliative  aid 

Stance  settings  describe  the  attributes  of  a  particular  stance.  These  were  applied  to  stance  Secure/Patrol. 
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Factor 

Experimental 

Range 

Description 

Stance  Attacks  Unit 

0.24-0.36 

Proportion  of  time  in  the  turn  spent  attacking 

Stance  Protect  Population 

0.32-0.48 

Proportion  of  time  in  the  turn  spent  protecting 

Stance  Provide  Aid 

0.16-0.24 

Proportion  of  time  in  turn  providing  aid 

Stance  Modify  Perception 

0.32-0.48 

The  level  to  which  the  population  will  be  reassured 
or  intimidated  by  the  Unit’s  actions 

Stance  Extort 

0-0.3 

Proportion  of  time  which  a  unit  extorts 

Stance  Counter  Crime 

0.24-0.36 

Proportion  of  time  which  a  unit  counters  crime 

Stance  Intel  Gather 

0.4-0. 6 

Level  which  a  unit’s  actions  allow  it  to  gather  intel 

Stance  QRF 

0.32-0.48 

Proportion  of  time  which  a  unit  is  on  QRF 

Stance  Average  Size 

24-36 

Average  size  of  unit  on  stance 

Stance  Protection  Modifier 

0.64-0.96 

Modifier  to  units  protection  value  while  on  stance 

Stance  Mobility  Modifier 

0.8-1. 2 

Modifier  to  units  mobility  value  while  on  stance 

Stance  Detectability  Modifier 

0.96-1.44 

Modifier  to  units  detectability  value  while  on  stance 

Stance  Detection  Modifier 

1 .2-1.8 

Modifier  to  units  detection  value  while  on  stance 

Stance  Recognition  Modifier 

0.8-1. 2 

Modifier  to  units  recognition  value  while  on  stance 

Settings  that  control  the  generation  of  the  population.  These  are  applied  to  the  entire  population 

Population  Decision  Radius 

40-60 

Distance  in  km  that  an  agent  will  look  across  to  find 
a  better  location 

Population  Memory  Coefficient 

2.4-3. 6 

Half  life  (in  turns)  of  the  current  consent  value 

Population  Consent  Political 
MScaler 

-0.64-0.96 

Controls  the  effect  of  ideology  differences  on 

Consent  (gradient) 

Population  Consent  Political 
CScaler 

0.08-0.12 

Controls  the  effect  of  ideology  differences  on 

Consent  (intercept) 

Population  Average  Term  In 
Prison 

2.4-3. 6 

Average  length  (in  turns)  of  a  custodial  sentence 

Population  Self  Presenters 

0.4-0. 6 

%  of  a  population  that  will  self-present  at  a 
hospital-outbreak 

Population  Police  Clear  Rate 

0.2-0. 3 

Gradient  of  the  police  clear  rate 

Population  Infection  MargGains 

0.16-0.24 

Related  to  the  infection  variable 

The  following  are  time  variables  used  in  the  PSOM  queuing  algorithm 

Combat  Mod  Que  Decay  Rate 

0.012-0.018 

Intelligence  decay  rate 

Percent  Force  on  Duty 

0.24-0.36 

Proportion  of  a  military  unit  on  duty  at  any  one  time 

Planning  Delay 

4. 8-7.2 

Length  of  time  in  hours  that  it  takes  to  plan 
operation 

Operation  Time 

3. 2-4.8 

Length  of  time  in  hours  that  it  takes  to  carry  out  a 
small  operation 
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Factor 

Experimental 

Range 

Description 

Recuperation  Time 

8-12 

Length  of  time  in  hours  the  portion  of  a  unit  in 
contact  should  be  unavailable 

Avg  Distance  Traveled 

6.4-9. 6 

Average  distance  that  a  force  will  have  to  move 
within  a  square  to  prosecute  a  contact 

Max  Fatigue 

0.64-0.96 

Unused 

Fatigue  Drop  Off  Factor 

3. 2-4.8 

Unused 

Force  Protection  Mean 

2.4-3. 6 

The  mean  of  the  force  protection  scale 

Force  Protection  KValue 

0.24-0.36 

Calibration  factor 

Force  Protection  Mod 

0.24-0.36 

Level  to  which  force  protection  actually  impacts 
casualties 

Mean  ROE 

2. 4-3. 6 

Mean  of  the  ROE  scale 

ROE  KValue 

0.24-0.36 

Calibration  factor 

ROE  Mod 

0.24-0.36 

Level  to  which  rules  of  engagement  actually  impact 
civilian  casualties 

Max  Leadership  Mod 

1. 6-2.4 

Maximum  modifier  possible  with  oo  leadership 

Min  Leadership  Mod 

0.16-0.24 

Minimum  modifier  possible  with  no  leadership 

Leadership  Drop  Off  Factor 

0.016-0.024 

Value  controlling  the  rate  at  which  the  value  of  the 
Leadership  modifier  curves 

Familiarization  Stranger 

0.4-0. 6 

Modifier  on  the  performance  gathering  of  a  person 
who  is  unfamiliar  with  his  surroundings 

Familiarization  Native 

1. 6-2.4 

Modifier  on  the  performance  gathering  of  a  person 
who  is  familiar  with  his  surroundings 

Familiarization  Learning 

0.08-0.12 

Control  on  the  curve  between  the  above  factors 

Experience  Conscript 

0.16-0.24 

Modifier  on  the  performance  of  a  new  recruit 

Experience  Vet 

1. 6-2.4 

Modifier  on  the  performance  of  an  experienced 
troop 

Experience  Learning  Factor 

0.0712- 

0.1068 

Control  on  the  curve  between  the  above  factors 

Inter  Unit  Base  Casualties  ATT 

0.08-0.12 

Level  of  casualties  caused  to  the  instigator  of  a 
contact 

Inter  Unit  Base  Casualties  DEF 

0.08-0.12 

Level  of  casualties  caused  to  the  defender  of  a 
contact 

Inter  Unit  Base  Contact  Size 

24-36 

Baseline  combined  size  of  the  attacking  and 
defending  forces  in  contact 

The  following  factors  are  relevant  to  each  particular  good. 

Goods  Expected 

0.4-0. 6 

Population  requirement  for  goods 

Good  Protection  Value  Power 

8000-12000 

Protection  score  of  power 

Good  Protection  Value  Water 

56-84 

Protection  score  of  water 
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Factor 

Experimental 

Range 

Description 

Good  Protection  Value 

Education 

56-84 

Protection  score  of  education 

Good  Protection  Value 
Healthcare 

120-180 

Protection  score  of  healthcare 

Table  6.  Factors  used  for  the  Settings  DOE  [From  (Parkman,  2008)] 


5.  Cumulative  Design  of  Experiments 

The  cumulative  experiment  is  the  aggregate  of  results  from  the  previous 
experiments’  analysis  and  a  desire  to  explore  new  factors  (see  Chapter  V).  After 
completing  the  very  first  test  run  the  concept  of  time  step  and  time  for  the  model  came 
into  question.  We  therefore  introduced  both  time  step  increment  and  overall  simulated 
time  as  factors  on  an  individual  experiment.  The  scenario  DOE  brought  significant 
insight  into  both  stances  and  marginal  gains  and  the  coalitions  limited  ability  to  increase 
consent.  We  therefore  focus  on  the  coalition  stances  in  this  design.  In  regards  to 
marginal  gains,  the  marginal  gain  values’  ranges  are  very  limited  in  this  DOE,  and  are 
lowered  to  allow  for  a  greater  change  in  consent.  Finally,  after  analysis  of  the  settings 
DOE,  further  analysis  of  the  unit  capabilities  was  required.  The  unit  attribute  values  are 
varied  over  a  greater  range  in  this  experiment  than  the  limited  20%  deviation  we  used 
earlier.  Table  7  explains  the  factors  used  in  this  DOE. 


FACTOR 

VALUES  and  RANGES 

Coalition  Stance 

Humanitarian  Aid  or  Secure  by  Patrol 

Time  Step 

7,  30,  60  in  days 

Sunni  MG  Security 

0. 1-0.2 

Sunni  Marginal  Gains 

0. 1-0.2 

Unit  Fire  Power 

50-150 

Unit  Protection 

50-150 

Unit  Sensors 

50-150 
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FACTOR 

VALUES  and  RANGES 

Unit  Size 

50-150 

Unit  Change  Attitude  Per  Man 

1-10 

Unit  Intelligence  Ability 

20-50 

Table  7.  Factors  used  for  the  Cumulative  DOE 


C.  EXPERIMENTAL  DEVELOPMENT  AND  TOOLS 

1.  Software  Upgrades  used  to  Expand  PSOM 

It  is  important  to  understand  that  PSOM  is  a  wargame.  It  was  developed  to 
explore  irregular  warfare  with  human  interaction.  Unfortunately,  this  setup  does  not 
allow  for  quick  turnover  analysis.  Adjusting  thousands  of  parameters  by  hand  and  then 
executing  twelve  months  of  combat  would  lead  to  potential  human  error  and  an 
unfathomable  amount  of  time.  In  2008,  when  the  Naval  Postgraduate  School  first 
received  PSOM  2.2.3,  the  batch  mode  was  not  flexible  enough  for  large  scale 
experimentation,  and  there  was  no  way  to  interface  PSOM  with  the  current  host  data 
farming  tools.  Such  an  experiment  had  never  been  done.  SEED  center  research  associate 
Adam  Larson  created  the  tools  needed  to  allow  parallel  runs  of  the  simulation  on  a  cluster 
of  computers.  These  tools  include  a  command  line  batch  mode  for  PSOM  allowing  the 
simulation  to  run  based  on  predetermined  settings  completely  independent  of  the 
graphical  user  interface  and  a  script  converter  which  converts  the  scenario  and  settings 
data  files  to  an  XML  format  compatible  with  current  data  farming  tools.  Additional 
changes  include  the  addition  of  direct  control  over  the  simulation’s  random  seed  allowing 
the  reproduction  of  any  particular  run.  Also,  in  order  to  allow  for  data  extraction,  we 
developed  a  PSOM  post  processor  that  pulls  the  non-normalized  data  for  each  individual 
run,  transforms  the  data  to  the  PSOM  scale  of  1  to  10,  takes  the  mean  of  multiple  runs  if 
used,  and  determines  the  variance  across  the  regions. 
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2.  Designs  of  Experiment  Used 


A  tremendous  amount  of  insight  can  be  obtained  in  a  very  efficient  manner  with  a 
well  designed  experiment.  In  this  thesis,  as  is  often  the  case  with  design  of  experiment, 
the  experiment  is  being  used  to  primarily  determine  which  factors  are  truly  important 
(Sanchez,  2008).  To  ensure  a  comprehensive  analysis,  the  experimental  design  should 
look  at  a  widely  ranging  combination  of  factors  while  at  the  same  time  trying  to  avoid 
confounding  to  ensure  causality  can  be  explained.  It  is  important  for  the  reader  to 
understand  which  design  techniques  were  used  throughout  this  study.  However,  these 
techniques  are  tools  for  analysis  and  not  a  subject  of  this  thesis;  therefore,  they  are  only 
briefly  explained  in  this  chapter. 

For  the  exploration  of  the  scenario  file,  this  thesis  uses  three  techniques  to 
construct  an  intelligent  design  of  experiments.  To  account  for  the  categorical  variables 
we  used  an  mk  factorial  design.  This  design  consists  of: 

4(Coalition  Stances)  x3(Sunni  Nationalist  Stances)  x2(IGO  Stances)  x2(Coalition  to  Sunni  Relations)=48  design  points 

A  nearly  orthogonal  Latin  hypercube  (NOLH)  DOE  is  used  for  the  remaining 
fourteen  factors.  A  design  is  nearly  orthogonal  if  the  maximum  absolute  pairwise 
correlation  between  any  two  input  columns  is  less  than  .05.  Latin  hypercubes  (LHs) 
provide  a  flexible  way  of  constructing  efficient  designs  for  multiple  quantitative  factors 
(McKay  et  al.  1979).  However,  due  to  an  inherent  randomness  in  their  construction,  they 
can  exhibit  substantial  correlations  among  the  input  variables — thus,  inhibiting  many 
statistical  procedures  with  which  we  would  like  to  use  to  analyze  relationships  between 
input  and  output  variables.  While  specially  constructed  orthogonal  LHs  exist  (see,  for 
example,  Ye  1998),  they  often  have  poor  space-filling  properties.  A  design  with  good 
space-filling  properties  is  one  in  which  the  design  points  are  scattered  throughout  the 
experimental  region. 

To  address  the  dearth  of  space-filling  orthogonal  LHs,  Cioppa  (2002)  used  a 
computationally  intense  heuristic  algorithm  to  generate  and  catalogue  a  set  of  NOLHs 
with  good  space-filling  properties.  These  flexible  designs  allow  for  the  efficient 
examination  of  many  factors  and  their  complex  relationships  with  very  low 
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correlations — thereby  facilitating  powerful  statistical  analysis  (Cioppa  &  Lucas,  2007). 
For  example,  by  using  Professor  Susan  Sanchez’s  NOLH  spreadsheet  implementation  of 
Cioppa’s  designs  (see  http://harvest.nps.edu  for  the  spreadsheet),  a  DOE  with  sixty-five 
design  points  is  sufficient  to  analyze  the  previously  mentioned  fourteen  continuous 
factors  throughout  their  entire  ranges.  A  basic  2k  factorial  design  requires  16,384  design 
points  and  would  only  explore  the  extreme  points  of  the  factors.  Finally,  by  using  a 
Cartesian  join  to  create  a  cross  design  of  the  full  factorial  design  with  the  NOLH  design, 
a  DOE  with  3,120  design  points  covers  a  plethora  of  possible  combinations  of  the  factors 
between  the  two  designs,  greatly  reducing  computing  expense. 

Each  design  point  (also  called  an  excursion)  represents  an  individual  run  of  the 
experiment  at  the  distinctive  parameter  values.  As  discussed  in  the  following  chapters  we 
found  little  variance  in  PSOM  output;  however,  because  of  the  slight  stochastic  nature  of 
PSOM,  each  excursion  was  run  five  times  with  a  different  random  number  seed  to 
account  for  variation  due  to  randomness  in  the  simulation  for  a  total  of  15,600  runs  on  the 
NPS  SEED  Center’s  cluster  of  fourteen  processors.  A  twelve  time  step  excursion  takes 
approximately  one  minute  to  run  on  a  standard  computer.  This  process  took 
approximately  fifteen  hours  of  computing  time  or  about  200  hours  of  processor  time. 

The  settings  design  of  experiment  was  driven  by  three  separate  characteristics  of 
the  parameters.  First,  all  data  was  continuous  in  nature,  so  the  NOLH  seemed  ideally 
suited.  However,  the  NOLH’s  described  by  Cioppa  (2002)  were  limited  to  twenty-nine 
factors  and  we  wanted  to  explore  sixty-six.  Fortunately,  COL  Alejandro  Hernandez  had 
developed  a  tool  for  his  PhD  dissertation,  Breaking  Barriers  to  Design  Dimensions  in 
Nearly  Orthogonal  Latin  Hypercubes  (Hernandez,  2008),  which  extends  the  limits  of  the 
NOLH  far  past  the  sixty-six  we  required.  Finally,  many  of  the  settings  values  are  not 
bounded  nor  well  described  in  the  PSOM  draft  documentation.  Therefore,  we 
manipulated  original  settings  factors  used  by  DSTL  by  ±  20%  to  mitigate  any  parameter 
dominance.  The  result  is  a  design  of  experiment  with  1,000  distinct  design  points 
thoroughly  covering  the  ranges  of  the  parameters.  Each  excursion  was  run  ten  times  (by 
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now  we  had  seen  PSOM  ran  rather  quickly)  for  a  total  of  10,000  scenario  runs.  This 
process  took  approximately  twenty-four  hours  to  complete  on  the  fourteen  processor 
cluster. 

The  cumulative  design  of  experiment  consists  of  only  ten  factors,  three  of  which 
are  categorical  while  the  remaining  seven  are  continuous.  For  this  experiment,  we 
crossed  a  stacked  NOLH  for  the  seven  continuous  factors  with  a  full  factorial  for  the 
categorical  variables,  resulting  in  792  design  points.  Each  design  point  was  run  ten  times 
for  a  total  of  7,920  runs. 

3.  Data  Mining  and  Meta  Models 

Data  mining  is  defined  by  Hand,  Mannila,  and  Smyth  “as  the  analysis  of 
observational  data  sets  to  find  unsuspected  relationships  and  to  summarize  the  data  in 
novel  ways  that  are  both  understandable  and  useful  to  the  data  owner”  (Hand,  2001).  The 
primary  approach  used  for  assessing  PSOM  once  the  data  is  “grown”  is  the  use  of  meta 
models.  This  study  followed  Kleijnen  and  Sargent’s  procedure  for  fitting  the  meta  model 
(Sargent,  2000).  A  meta  model  is  a  an  approximation  of  the  input  and  output  data  within 
a  simulation  (Sargent,  2000).  It  is  merely  a  simple  function  used  to  approximate  a  very 
complicated  simulation. 

By  simplifying  the  simulation  we  can  gain  tremendous  insight  into  the  actual 
nature  of  PSOM.  For  example,  this  study  uses  tools  such  as  polynomial  regression 
models  to  simplify  the  very  complex  inner  workings  of  PSOM.  Because  the  meta  models 
used  are  not  meant  as  prediction  models,  but  rather  to  gain  inference  on  the  model,  the 
meta  models  overall  accuracy  is  not  the  overall  goal,  but  gaining  insight  is.  The  primary 
types  of  meta  models  used  are  least  squares  regression  and  logistic  regression.  For  much 
of  the  validity  measures  of  the  meta  model,  absolute  relative  error  and  R-Squared  are  the 
primary  measures  used  in  determining  usefulness.  We  also  used  neural  networks  as  a 
second  modeling  technique  to  merely  validate  the  regression  models.  We  assess  the 
significance  of  the  regressors  in  the  meta  model  and  reduce  the  overall  number  of 
regressors  using  the  concept  of  Occam’s  Razor;  remembering  that  although  we  can 
closely  fit  the  model  and  minimize  error,  we  can  lose  sight  on  which  parameters  are  truly 
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influential  to  PSOM.  After  the  meta  model  has  shown  light  on  key  factors,  further 
analysis  into  the  raw  data  and  the  mathematical  algorithms  within  PSOM  can  be 
conducted. 
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V.  DATA  ANALYSIS 


Information  is  data  that  has  been  given  meaning  by  way  of  relational 
connection. 


(Ackoff,  1989) 

By  leveraging  design  of  experiment  with  powerful  data  farming  tools  we  are 
provided  a  tremendous  amount  of  data  which  is  not  yet  tractable  for  a  simulation  that  has 
no  true  closed  form  solution.  Although  much  of  the  “heavy  lifting”  is  complete,  we  still 
need  to  put  the  tremendous  amount  of  data  into  a  manageable  form  and  quantitatively 
describe  its  relevance.  The  purpose  of  this  chapter  is  to  describe  the  processes  used  to 
analyze  the  data.  This  analysis  is  formatted  around  gaining  insight  into  the  original 
questions  posed  in  the  beginning  of  this  thesis.  We  demonstrate  that  by  using  data 
models  and  algorithmic  models  (neural  networks)  in  conjunction  with  our  experimental 
designs,  valuable  conclusions  can  be  gained  about  very  complex  simulation  models.  At 
the  conclusion  of  this  process,  key  points  about  the  nature  of  PSOM  are  presented.  These 
conclusions  are  discussed  in  detail  in  Chapter  VI,  less  the  detailed  mathematical  analysis. 

A.  DATA  COLLECTION 

In  line  with  its  purpose  as  a  human-in-the-loop  wargame,  the  output  for  PSOM  is 
primarily  graphic  in  nature  and  meant  to  be  displayed  on  a  map  of  the  area  of  operations. 
These  briefing  maps  are  color  coded,  and  for  the  MOEs  security  and  consent  range  from 
0  to  10  (see  Figure  11).  Fortunately,  PSOM  does  have  the  ability  to  output  data  by  grid 
square  across  the  area  of  concern.  These  data  are  output  into  a  comma  separated  value 
file  (CSV)  in  their  raw  form  without  having  been  transformed  to  the  fore  mentioned 
normalized  scale.  Each  CSV  file  contains  the  output  for  an  individual  MOE  by  faction  or 
ethnic  group  in  a  single  run.  Therefore,  each  design  of  experiment  provides  over  100,000 
output  files.  Because  the  primary  emphasis  of  this  study  is  limited  to  the  Sunni  consent 
and  security  MOEs,  the  data  concerning  these  values  is  an  aggregate  of  the  mean 
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response  of  each  region  based  on  coalition  forces’  areas  of  responsibility  and  the  entire 
country.  The  following  example  describes  the  mean  consent  values  of  the  Sunni  toward 
the  coalition  forces  across  all  of  Iraq, 


MeanSunniConsentCoalUionIraq 


y ;  SunniConsent  Co 


Where  k  =  the  total  run  number  within  the  design  point. 

«=the  number  of  grids  squares  in  the  scenario. 

A  similar  method  was  used  to  look  at  the  level  of  variance  throughout  each  AOR 
and  the  entire  country. 


SunniConsentVariancecoaMonIraq 


I  I  ( Xq  ■  X coalition, iraq  ) 

(£*«)-  1 


Where  x  is  the  consent  for  coalition  in  grid  square  i  on  run  j  and  n,  the  total 
number  of  grid  squares,  is  multiplied  by  k,  the  number  of  runs  of  the  design  point.  This 
value  provides  insight  into  the  overall  volatile  nature  of  the  responses  across  the  area  of 
concern. 


This  technique  allows  for  further  analysis  into  the  six  regions  of  the  country 
without  the  overwhelming  resolution  of  135  responses.  This  permits  significant  enough 
granularity  in  the  outcomes  to  raise  a  flag  when  outcomes  are  interesting,  ensuring  further 
analysis  when  needed.  At  this  point,  either  analysis  can  take  place  on  specific  region, 
individual  grid  squares,  or  the  game  can  be  played  and  observed  with  the  interesting 
parameters  from  the  specific  corresponding  design  point. 


Arguably  this  technique  is  not  ideal.  Measures  of  effectiveness  are  measured  by 
geographic  location  and  not  by  population.  Therefore,  there  is  no  appropriate  weighting 
for  population  size  in  a  particular  area.  For  example,  region  4,  which  is  the  highly 
populated  city  of  Baghdad,  consists  of  only  for  two  squares,  which  is  the  same  number  of 
squares  as  Samarra,  which  only  has  a  10%  of  the  population  of  Baghdad.  This  is  an 
acknowledged  weakness  in  this  study;  however  specific  cells  and  regions  were  addressed 
when  required  (we  played  over  100  games  step-by-step  throughout  this  process). 
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Initially,  this  entire  process  was  conducted  on  a  small  scale  design.  This  provided 
initial  insight  into  the  stochastic  nature  of  PSOM,  and  some  initial  factor  analysis  as  well 
as  verifying  the  functionality  of  the  computational  tools  developed.  Although  the  initial 
experiment  was  meant  to  be  a  test,  some  significant  insight  was  gained.  The  focus  of  this 
design  was  to  demonstrate  the  significance  of  time  step  and  faction  stance.  The  factors  in 
this  experiment  are  displayed  in  Table  8.  In  this  case,  each  simulation  was  run  for  12 
time  steps.  So  the  simulation  was  two  years,  one  year,  or  12  weeks  in  simulation  time. 


IGO  Stance 

Coalition 

Stance 

Sunni 

Nationalist 

Stance 

Time  Step 

Level  1 

Active 

Providing  Aid 

Providing 

7  Days 

Level  2 

Withdrawn 

Attacking 

Attacking 

30  Days 

Level  3 

Securing 

60  Days 

Table  8.  Initial  Test  DOE  Factors 


Each  design  point  was  run  30  times  to  ensure  variance  could  be  accurately 
estimated.  The  results  show  that  although  variance  exists  between  each  of  the  30 
iterations,  it  is  minimal.  Table  9  shows  the  quantiles  of  mean  Security  and  variance  by 
grid  squares  for  a  randomly  drawn  excursion  file.  It  is  important  to  note  is  that  although 
the  security  values  cover  a  large  portion  of  the  1  to  10  domain  the  variance  between  the 
30  iterations  by  grid  square  is  minimal.  Consent  results  are  similar. 
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Quantiles  for 
Gridsquare 

N=135 

Mean  Security  by 

Quantiles  for  Security  Variance  (a-2)  of  30 

runs  by  GridSquare  N=135 

Quantiles 

Quantiles 

100.0% 

maximum 

9.5050 

100.0% 

maximum 

0.23809 

99.5% 

9.5050 

99.5% 

0.23809 

97.5% 

8.2720 

97.5% 

0.03529 

90.0% 

8.0924 

90.0% 

0.00209 

75.0% 

quartile 

8.0787 

75.0% 

quartile 

0.00200 

50.0% 

median 

7.5153 

50.0% 

median 

0.00166 

25.0% 

quartile 

5.3991 

25.0% 

quartile 

0.00067 

10.0% 

5.3754 

10.0% 

0.00041 

2.5% 

3.9384 

2.5% 

0.00034 

0.5% 

2.1749 

0.5% 

0.00031 

0.0% 

minimum 

2.1749 

0.0% 

minimum 

0.00031 

Table  9.  Looking  at  the  Means  and  Variance  of  PSOM  Output  for  Security 


The  largest  variance  of  .24  (Standard  Deviation  =  .48)  is  from  grid  location  (9,7), 
and  although  this  value  is  large  in  comparison  to  the  rest  it  is  5%  of  the  overall  range 
covered  and  arguably  negligible  when  looking  at  the  entire  country  of  135  squares. 

Using  a  least  squares  quadratic  linear  regression  with  two-way  interactions  built 
by  stepwise  regression  we  created  a  meta  model  to  determine  the  significance  of  the  fore 
mentioned  factors  in  respect  to  the  response  variable  Iraq  mean  security.  Our  primary 
concern  in  this  analysis  is  to  ensure  that  12  time  steps,  whether  they  equate  to  2  years  or  3 
months  do  not  provide  similar  results.  The  scaled  estimate  graph  in  Figure  12  shows  the 
least  square  regression  meta  model  and  that  time  step  is  indeed  the  most  significant  factor 
of  the  three  (stances  are  further  analyzed  in  the  following  sections).  The  model  shows 
that  the  7-day  time  step  has  the  most  significant  affect  on  security  followed  by  the  30-day 
and  60-day  time  step.  This  provides  more  knowledge,  in  that  as  expected,  the  longer 
units  are  active  in  a  particular  area  the  less  marginal  impact  they  have  on  security.  The 
model  demonstrates  the  potential  to  reflect  a  point  of  diminishing  returns. 
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Scaled  Estimates  of  the  Regression  resulting  in  an  R-Squared  of  .82  and  Root  Mean 
Squared  Error  of  .41 


Scaled  Estimates 

Continuous  factors  centered  by  mean,  scaled  by  range/2 

Scaled 


Term 

Estimate 

Intercept 

6.5631114 

Time  Step{7-30&60} 

-0.647394 

Time  Step{30-60} 

-0.283486 

Coalition  Stance{2&1-0} 

-0.312023 

Coalition  Stance{2-1} 

-0.121373 

Sunni  Stance{1-0&8} 

-0.382602 

Sunni  Stance{0-8} 

-0.180473 

(Time  Step{7-30&60}+0.33333)*Coalition  Stance{2-1} 

0.3298105 

Std  Error  t  Ratio  Prob>|t| 


0.027843 

235.72 

<0001* 

0.029532 

-21.92 

<0001* 

0.034101 

-8.31 

<0001* 

1 

0.029532 

-10.57 

<0001* 

1 

0.034101 

-3.56 

0.0005* 

0.029532 

-12.96 

<0001* 

| 

0.034101 

-5.29 

<0001* 

| 

0.036169 

9.12 

<.0001* 

Figure  12.  Explanation  of  the  Test  DOE  Regression  Model 


Analyzing  the  data  by  the  average  security  across  Iraq  against  time  step  from 
PSOM  we  can  see  the  significance  of  time  step.  Figure  13  shows  the  results  of  a 
Kruskal- Wallis  Test  comparison  of  security  in  regard  to  time  step  and  a  quantile  plot  of 
security  by  time  step.  The  Kruskal- Wallis  Test  is  a  non  parametric  test  for  comparing 
multiple  populations  where  the  null  hypothesis  is  the  populations  are  equivalent.  First, 
looking  at  the  P- Value  for  the  hypothesis  test  there  is  a  statistically  significant  difference 
between  security  when  subdivided  by  time  step.  The  quantile  plot  visually  depicts  the 
test  results.  Figure  13  shows  the  substantial  difference  in  security  between  a  7-day  time 
step  and  a  30-day  time  step  and  just,  as  the  meta  model  states,  the  difference  is  not  as 
significant  between  the  30-day  and  60-day  time  step.  For  a  fixed  number  of  steps,  larger 
time  steps  results  in  greater  security  in  this  scenario. 
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Figure  13.  Difference  in  Security  Determined  by  Time  Step 


B.  SCENARIO  SPECIFIC  DESIGN  OF  EXPERIMENT 

The  scenario  specific  DOE  covers  those  factors  which  are  either  underlying 
assumptions  in  the  specific  scenario  or  can  be  inputted  by  the  players  themselves  as 
discussed  in  Chapter  IV. 

1.  Consent 

Consent  is  defined  as  the  degree  to  which  the  population  supports  and  submits  to 
the  will  of  a  faction.  It  is  a  result  of  the  difference  between  expectations  of  an  ethnic 
group  and  the  actual  provision  (including  security)  generated  by  a  particular  faction. 
Figure  14  is  a  correlation  matrix  which  substantiates  the  use  of  the  mean  overall  Sunni 
consent  for  coalition  forces.  The  mean  overall  consent  and  the  consent  of  each  sub  region 
show  a  very  strong  positive  correlation. 


Variable 

By  Varaible 

Correlation 

Significance  Prob 

SUNNI  AVG  CoalitionConsent 

Region  1  Mean 

0.98 

0 

SUNNI  AVG  CoalitionConsent 

Region  2  Mean 

0.99 

0 

SUNNI  AVG  CoalitionConsent 

Region  3  Mean 

0.99 

0 

SUNNI  AVG  CoalitionConsent 

Region  4  Mean 

0.90 

0 

SUNNI  AVG  CoalitionConsent 

Region  5  Mean 

0.98 

0 
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Figure  14.  Correlation  of  Mean  Consent  by  Region  and  Mean  Consent  of  Iraq  for  the 

Sunni  Population  Agents  toward  the  Coalition 

Worth  noting  is  the  decreased  correlation  between  the  overall  average  consent  in 
Iraq  and  that  of  Region  4.  Region  4  is  Baghdad,  which  due  to  its  large  population  and 
larger  amount  of  coalition  units  has  a  bit  more  extreme  activity.  Region  4  consent  is 
isolated  and  analyzed  later  in  this  chapter. 

Because  initial  consent  is  a  factor  in  the  experiment,  the  meta  model  response  is 
the  difference  between  the  beginning  and  the  end  of  the  simulation  run  ( Consent  Final- 
Consent  Initial).  An  initial  look  at  this  response  variable  (Figure  15),  shows  that 
regardless  of  factor  settings  mean  Sunni  consent  for  the  coalition  did  not  increase  for  any 
of  the  3,120  design  points.  This  was  a  troublesome  point  throughout  the  initial 
experiment.  Consent  toward  a  faction  is  a  function  of  the  goods  created,  to  include 
security,  by  a  particular  faction  and  the  corresponding  marginal  gains  of  the  population 
agents.  It  took  multiple  iterations  of  the  design  process  to  actually  create  goods  because 
goods  production  is  a  balance  of  infrastructure  produced  and  human  capital  produced. 
Once  this  balance  is  met,  which  it  was  for  this  DOE,  production  increased. 
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Difference  in  Consent 


-8  -7  -6  -5  -4  -3  -2  -1  0 


Quantiles 


100.0% 

maximum 

-0.135 

99.5% 

-0.536 

97.5% 

-0.715 

90.0% 

-1.245 

75.0% 

quartile 

-2.108 

50.0% 

median 

-3.444 

25.0% 

quartile 

-6.069 

10.0% 

-7.819 

2.5% 

-8.455 

0.5% 

-8.699 

0.0% 

minimum 

-8.699 

Figure  15.  Histogram  and  Quantiles  of  the  Difference  in  Initial  Sunni  Consent  and 

Final  Sunni  Consent 

The  next  step  is  to  create  a  meta  model  to  gain  some  insight  into  what  factors  are 
contributing  to  the  change  in  Sunni  consent  toward  the  coalition.  This  meta  model  is  a 
quadratic  fit  least  squares  model  allowing  for  two-way  interactions.  Stepwise  regression 
in  JMP  7  resulted  in  an  adequate  model  introducing  relatively  few  factors.  Figure  16  is 
comprised  of  three  separate  figures.  The  summary  of  fit  shows  an  R-Square  of  .947, 
which  is  a  statistic  describing  the  proportion  of  the  variance  accounted  for  by  the  meta 
model.  Because  of  the  nature  of  the  R-Square  statistic  this  study  carefully  considers  the 
number  of  factors  in  the  model  in  conjunction  with  a  particular  R-Square  with  the  goal 
being  to  explain  as  much  variation  in  the  model  as  possible  with  a  minimum  number  of 
factors.  Figure  16  shows  that  with  an  R-square  of  .94  and  a  mean  square  error  of  .35  our 
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model  is  clearly  significant.  The  actual  by  predicted  plot  shows  a  fairly  linear 
relationship,  suggesting  the  meta  model  is  sufficient  for  analysis  of  these  factors  and  this 
response. 


Summary  of  Fit 

RSquare 

0.938032 

RSquare  Adj 

0.937792 

Root  Mean  Square  Error 

0.588595 

Mean  of  Response 

-4.03833 

lobservations  (or  Sum  Wqts) 

3120 

Analysis  of  Variance 

Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Model 

12 

16293.807 

1357.82 

3919.2971 

Error 

3107 

1076.402 

0.35 

Prob  >  F 

C.  Total 

3119 

17370.208 

0.0000*1 

Actual  by  Predicted  Plot 


1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 11 1 1 1 1 1 

-8.0  -7.0  -6.0  -5.0  -4.0  -2.0  -1.0  0.0 


Difference  in  Consent  Predicted 
P<.0001  RSq=0.95  RMSE=0.5412 


Figure  16.  Summary  Information  of  Consent  Meta  Model 


Figure  17  displays  the  factors  introduced  into  the  meta  model  (less  the  intercept). 
The  effects  are  sorted  by  the  absolute  value  of  the  t-ratio,  showing  the  most  significant 
effects  at  the  top.  A  bar  graph  shows  the  t-ratio,  with  a  line  showing  the  0.05  significance 
level  (JMP  7  Help). 


Sorted  Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Normalized  Inititial  Consent 

-0.72856 

0.004255 

-171.2 

Sunni  MG  Security 

-11.53384 

0.119816 

-96.26 

(Sunni  MG  Security-0.45003)*( Normalized  Inititial  Consent-5) 

-3.274689 

0.04548 

-72.00 

(Sunni  MG  Security-0.45003)*(Sunni  MG  Security-0.45003) 

81.535595 

1.560473 

52.25 

Coalition  Stance{1&2&8-0} 

-0.260095 

0.012356 

-21.05 

(Sunni  MG  Security-0.45003)*(Sunni  Marginal  Gains-0.45003) 

32.613802 

1.726586 

18.89 

Sunni  Marginal  Gains 

-1.81181 

0.119842 

-15.12 

(Coalition  Force  Protection-3.0061 5)*(Sunni  Force  Protection-3.00615) 

-0.110728 

0.009533 

-11.62 

Coalition  Force  Protection 

-0.088272 

0.008994 

-9.81 

Sunni  Force  Protection 

0.0759253 

0.008993 

8.44 

Coalition  Stance{1-2&8} 

-0.094726 

0.012906 

-7.34 

(Coalition  Stancell -2&8)+0.25)*(Sunni  Marqinal  Gains-0.45003) 

0.8352763 

0.144499 

5.78 

Prob>|t| 


0.0000* 

0.0000* 

0.0000* 

0.0000* 

<0001* 

<0001* 

<0001* 

<0001* 

<0001* 

<0001* 

<0001* 

<0001* 


Figure  17.  Scenario  DOE  Sorted  Parameter  Meta  Model 


For  adjudication  of  our  model,  we  also  use  the  neural  network  variable 
importance  tool  from  the  Clementine  software  package.  Figure  18  is  developed  using  a 
Bayesian  Neural  Network.  This  technique  is  used  throughout  the  thesis  to  ensure  that  we 
are  focused  on  the  significant  factors.  As  can  be  seen  by  comparing  Figures  16  and  17,  in 
the  two  completely  different  modeling  types  the  same  factors  (marginal  gains,  initial 
consent,  and  coalition  stance)  are  significant. 
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Variable  Importance 

Targets:  Overall_AVG_Sunni_Consent 


Sunni  MG  Security- 
Normalized  Inititial  Consent- 
Sunni  Marginal  Gains- 
Coalition  Stance' 
Sunni  AIF  Stance- 
Coalition  Reputation- 
Coalition  ROE' 
Sunni  Force  Protection- 
Coalition  Leadership- 
Coalition  Cas  Tolerance- 
Coalition  Force  Protection- 
Coalition  Experience- _ 
Coalition  Turns  at  location- 
Sunni  AIF  ATK  US- 
Initial  Threat- 
Sunni  ROE-T 
Sunni  AIF  ATK  Iraqi  Govt-] 
Sunni  Political  Ideology- 
NGO  Stance- 
Coalition  intel  Relationship  with  ...- 


ationShip  with  Sunni  AIF 


Isunni  MG  S 

±=rp 


Figure  18.  Variable  Importance  for  the  Response  Consent  from  the  Scenario  DOE 
found  from  Using  a  Bayesian  Neural  Network  Produced  Using  Clementine 


From  Figures  17  and  18  we  can  gain  many  insights  about  the  consent  MOE.  First 
off,  just  looking  at  the  factors  without  interactions,  the  initial  consent  is  the  most 
significant  factor  followed  by  the  Sunni  population’s  marginal  gain  for  security.  Sunni 
marginal  gain  for  security  also  enters  the  model  as  a  quadratic  term  indicating  its 
importance  to  the  model.  Arguably  this  makes  sense;  however,  these  are  both 
assumptions  made  in  the  scenario  file  which  appear  able  to  drive  the  game.  Coalition 
stance  is  a  significant  factor;  however,  it  is  not  nearly  as  influential  as  the  previously 
mentioned  assumptions.  Sunni  marginal  gains  for  goods  less  security  is  also  introduced 
as  a  significant  factor.  Finally,  faction  force  protection  values  show  significance. 

Looking  more  closely  into  the  relationship  between  initial  consent  and  the  Sunni 
marginal  gain  for  security  we  see  the  significance  of  these  two  factors  in  relationship  to 
the  final  average  consent  of  the  Sunni  population  to  the  coalition.  Figure  19  is  a  contour 
plot  with  the  Sunni  marginal  gain  for  security  on  the  x-axis,  normalized  consent  on  the  y- 
axis,  and  colors  from  red  to  blue  representing  the  corresponding  values  of  the  final 
average  Sunni  consent  toward  the  coalition  at  each  design  point. 
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Contour  Plot  for  SUNNI  AVG  CoalitionConsent 


l  I  l  I  l  l  l  l  l  l  l  |  l  l  l  l  l  l  l  l  l  |  l  l  l  l  l  l  l  l  l  |  l 


Sunni  MG  Security 


Figure  19.  Contour  Plot  of  Sunni  Marginal  Gain  for  Security  and  Initial  Consent  vs. 
Final  Sunni  Consent  toward  Coalition  [Best  viewed  in  color] 


This  is  very  interesting  because,  regardless  of  faction  settings,  if  the  Sunni  MG 
for  security  is  .5  or  greater  consent  for  the  coalition  will  fall  below  1  in  this  scenario.  We 
can  also  see  the  significance  of  the  initial  assumption  of  consent.  When  the  Sunni 
marginal  gain  for  security  is  set  at  a  relatively  low  value,  greater  initial  consent  results  in 
greater  final  consent.  Also  of  note  is  the  relative  jaggedness  of  the  contour  plot,  which  in 
part  is  caused  by  the  influence  of  other  factors. 

When  replaying  parts  of  the  game  individually  production  does  increase,  which  is 
directly  tied  to  consent.  This  leads  to  the  need  for  a  more  detailed  analysis  of  consent  in 
part  1(a)  in  this  section.  Consent  has  been  the  more  complicated  of  the  MOEs  to  look  at. 
In  PSOM  it  is  largely  a  function  of  production  level  of  the  goods  the  population  desires. 
It  requires  money,  manpower,  and  effort  put  toward  production.  However,  if  consent  is 
low,  it  is  difficult  to  hire  manpower. 

The  importance  of  the  assumptions  within  the  scenario  is  notable;  however, 
PSOM  is  a  wargame,  and  the  influence  the  players  have  over  consent  is  important. 
Figure  20  is  a  bar  graph  comparing  final  Sunni  consent  toward  the  coalition  in  regards  to 
Sunni  Nationalist  stance  and  coalition  stance.  The  graph  shows  that  regardless  of  Sunni 
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Nationalist  stance,  coalition  consent  can  reach  its  highest  points  when  its  stance  is  zero 
(providing  humanitarian  aid).  But  what  is  more  surprising  is  that  the  coalition  consent 
when  withdrawn  is  equivalent  regardless  of  Sunni  Nationalists  stance.  Just  as  surprising 


is  that  whether  the  coalition  is  providing  security  or  in  an  attacking  stance  we  see  about 
the  same  consent  levels. 


Figure  21  displays  the  consent  toward  coalition  in  regard  to  coalition  stance.  The 
green  diamond  provides  a  95%  confidence  interval  for  the  sample  mean  of  consent  at  the 
particular  stance.  Conducting  a  non  parametric  Kruskall  Wallis  test  to  determine  which 
stances  matter,  we  see  that  stance  0  (provide  humanitarian  aid)  has  a  significantly  higher 
overall  median  than  the  other  stances.  This  coincides  with  the  definition  of  consent. 
However,  the  consent  resulting  from  the  other  stances  shows  no  significant  difference. 
Therefore,  it  does  not  appear  to  matter  whether  all  the  coalition  forces  are  securing, 
attacking,  or  withdrawn,  which  is  surprising. 
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Oneway  Analysis  of  Overall_AVG_Sunni_Consent  By  Coalition  Stance 
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Figure  21 .  Means  Diamond  Plot  of  Sunni  Consent  toward  Coalition  in  Regards  to 
Coalition  Stance  and  a  Kruskal  Wallis  Test  for  Significances  in  the  Factor  Stance 


a.  Increasing  Consent  for  Coalition 

The  fact  that  the  average  consent  across  Iraq  never  increased  proves 

provoking  and  requires  further  attention.  Looking  at  the  separate  regions  of  Iraq  and  then 

at  the  actual  grid  squares  there  are  individual  grid  squares  where  consent  increases. 

Focusing  on  Region  4,  Baghdad,  this  becomes  evident.  There  are  118  of  the  3,120  total 

design  points  in  which  consent  toward  the  coalition  increased  in  Region  4.  Figure  22 

displays  the  histogram  of  change  of  consent  toward  the  coalition  in  the  region. 
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Figure  22.  Region  4  (Baghdad)  Difference  in  Initial  and  Final  Consent 


By  replaying  a  few  of  these  design  points  where  consent  increased  it 
became  evident  that  production  also  increased  primarily  in  Region  4.  Intuitively,  this 
means  that  the  coalition  stance  is  set  at  0  (humanitarian  aid). 
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A  nominal  logistic  regression  meta  model  depicts  the  probability  based  on 
particular  factors  that  the  Sunni  consent  value  in  Region  4  increases  or  decreases. 
Because  this  model  is  only  used  to  gain  inference  the  focus  is  only  on  the  factors 
currently  known  to  be  significant.  The  resulting  meta  model  paints  a  picture  of  which 
factors  are  important  to  providing  an  increase  in  consent.  From  the  parameter  estimates 
below  we  learn  that  a  coalition  stance  of  0  (humanitarian  aid),  combined  with  a  high 
Sunni  ROE  (restrictive)  and  low  Sunni  marginal  gains,  will  provide  a  high  probability  for 
increased  consent. 
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Figure  23.  Nominal  Logistic  Regression  Model  Statistics  for  whether  or  not  Consent 

Increased  or  Decreased  in  Baghdad 


The  prediction  profiler  in  JMP  7  is  an  interactive  tool  that  allows  analysis 
of  the  response  variable  (consent)  due  to  changes  in  the  model’s  regressors.  Figure  24 
depicts  the  prediction  profiler  for  the  nominal  logistic  regression  model  with  the  above 
factors.  The  values  in  the  y-axis  are  the  probabilities  of  consent  increase.  Figure  24a 
shows  that  if  the  Sunni  AIF  stance  is  1  (attack),  regardless  of  coalition  stance,  the 
probability  is  very  low  that  there  will  be  an  increase  in  consent.  Figure  24b  shows  the 
drastic  change  in  the  probability  of  consent  increase  when  the  Sunni  stance  is  changed  to 
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withdrawn  (all  other  factors  are  held  constant).  Figure  24c  increases  (tightens)  only 
Sunni  ROE,  which  in  turn  increases  the  probability  of  an  increase  in  consent.  Figure  24d 
shows  the  extreme  effect  that  changing  Sunni  marginal  gains  security  has  on  the  model. 
With  a  slight  increase  in  the  factor  the  probability  of  increasing  consent  is  marginal. 
Looking  at  Figure  24a  and  comparing  Sunni  marginal  gains  and  Sunni  marginal  gains  for 


security,  we  can  see  that  the  functions  are  similar  and  have  similar  effects  on  the  model. 


Figure  24.  Effects  Profiler  for  Logistic  Regression  Model  Describing  Probability 
Sunni  Consent  toward  Coalition  Increases  Region  4 
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Now,  looking  back  at  the  raw  data  for  the  change  in  Region  4  consent,  we 
see  that  it  matches  our  model.  We  primary  see  consent  increase  when  the  coalition 
provides  humanitarian  aid  while  the  Sunni  Nationalists  are  withdrawn.  Interestingly,  the 
second  best  combination  for  the  scenario  is  when  the  coalition  is  also  withdrawn  as 
shown  in  Figure  25. 


Figure  25.  Region  4  Change  in  Sunni  Consent  toward  the  Coalition  in  Regards  to 

Coalition  and  Sunni  Nationalist  Stance 


Figure  26  is  used  to  verify  the  accuracy  of  the  model  and  to  determine  if 
the  inference  is  correct  about  the  Sunni  marginal  gains  and  Sunni  marginal  gains  for 
security.  The  graphs  are  both  histograms  of  the  118  data  points  in  which  consent  for  the 
coalition  increased  in  Region  4.  These  same  charts,  if  they  were  displayed  using  the 
original  3,120  design  points,  would  be  uniform.  However,  these  histograms  both  tend 
toward  the  lower  values  of  the  factors.  It  is  apparent  that  if  a  scenario  is  to  be  functional 
in  the  consent  category,  these  values  should  be  set  relatively  low.  The  exceptions  to  this 
conclusion  are  easily  explainable  after  further  analysis.  In  regards  to  Sunni  marginal 
gains,  there  are  twelve  points  on  the  histogram  which  have  this  factor  at  a  higher  level. 
The  corresponding  marginal  gain  for  security  for  each  of  these  design  points  is  at  its 
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lowest  value  of  .3,  which  apparently  is  influential  enough  to  offset  the  marginal  gains. 
The  few  values  at  the  Sunni  marginal  gains  for  security  that  are  at  the  factor’s  higher 
level  have  the  lowest  level  of  initial  consent,  the  Sunni  Nationalists,  withdrawn,  and  a 
very  high  Sunni  ROE  value  which,  when  all  align,  allow  for  a  very  small  (.16  on  a  scale 
of  1  to  10)  increase  in  consent. 
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Figure  26.  Histograms  and  Quantiles  of  Marginal  Gains  Values  for  1 18  Points  in 

which  Consent  Increased  in  Bagdad 


b.  Consent  for  the  Iraqi  Government 

One  of  the  key  measures  of  a  legitimate  government  is  that  it  has  the 
consent  of  the  population  (U.S.  Army,  2006).  It  is  important  to  see  what  effects  the 
coalition  can  have  on  the  consent  of  the  Iraqi  government  remembering  that  all 
humanitarian  aid  by  the  coalition  targeted  the  Iraqi  government,  giving  the  Iraqi 
government  partial  credit  for  the  service. 
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A  first  look  at  the  data  (Figure  27)  shows  that  99%  of  the  time  consent 
dropped  (starting  value  was  3.6);  however,  we  were  able  to  increase  it  a  few  times.  It  is 
also  important  to  note  that  coalition  consent  often  was  driven  to  zero,  but  this  was  not  the 
case  for  the  government  as  shown  in  Figure  27. 


Figure  27.  Histogram  and  Quantiles  Sunni  Consent  Across  Iraq  toward  Iraqi 

Government  for  Scenario  DOE 


Figure  28  shows  the  results  of  a  quadratic  least  squares  regression  model 
allowing  for  two-way  interactions  with  the  response  consent  for  the  government.  The 
model  is  limited  to  three  factors:  Sunni  marginal  gains  security,  Sunni  marginal  gains, 
and  coalition  stance,  with  a  resulting  R-squared  value  of  0.91. 


69 


Summary  of  Fit 
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Figure  28.  Regression  Meta  Model  for  Sunni  Consent  for  the  Iraqi  Government 


Once  again,  Sunni  marginal  gains  for  security  prove  to  be  the  most 
significant  factor.  Looking  at  a  contour  plot  of  Sunni  marginal  gains  for  security  and 
Sunni  marginal  gains  (Figure  29),  there  is  a  strong  relationship  between  consent  for  the 
Iraqi  government  and  these  factors.  Sunni  MG  for  security  can  dominate  consent  for  the 
Iraqi  government  just  as  with  the  coalition. 


Sunni  MG  Security 


Figure  29.  Contour  Plot  of  Sunni  Marginal  Gain  for  Security  and  Sunni  Marginal 
Gains  for  Good  versus  Final  Consent  for  the  Iraqi  Government 
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The  red  line  in  Figure  30  is  a  quadratic  fit  of  government  consent  in 
regards  to  only  Sunni  marginal  gains  for  security.  This  factor  alone  accounts  for  87%  of 
the  model’s  variation. 


Figure  30.  Bivariate  Normal  Fit:  Sunni  Consent  toward  Iraqi  Government  (Response) 

and  Sunni  Marginal  Gains  Security  (Regressor) 

Figure  3 1  shows  a  bar  graph  of  Iraqi  government  consent  in  regards  to 
coalition  stance  and  the  corresponding  non  parametric  test  for  significance  of  the  stances. 
The  graph  shows  that  the  highest  values  of  Iraqi  consent  occur  when  the  coalition  stance 
is  0  (humanitarian  aid).  This  reflects  a  conscious  decision  in  the  scenario  setup  to  credit 
the  Iraqi  government  with  coalition  production  proving  this  aspect  of  the  game  is 
effective.  We  also  see  that  when  the  coalition  is  providing  security  (level  2)  we  are  able 
to  get  a  slightly  higher  consent  toward  the  Iraqi  government  than  when  attacking  (level  1) 
or  withdrawn  (level  8),  which  makes  sense  as  well.  Based  on  the  non  parametric  test 
there  is  a  statistically  significant  difference  among  the  stances  in  regard  to  consent. 
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Oneway  Analysis  of  Consent  for  Iraqi  Gov't  By  Coalition  Stance 
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Figure  3 1 .  Iraqi  Government  Consent  in  Regards  to  Coalition  Stance  and  the 
Corresponding  Non  Parametric  Test  for  Significance  of  the  Stances 


The  correlation  between  the  change  in  coalition  consent  and  Iraqi 
government  consent  is  shown  in  Figure  32.  It  is  substantial,  which  is  logical,  as  the 
coalition  and  the  Iraqi  government  are  allies  in  this  scenario. 
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Figure  32.  The  Correlation  between  the  Change  in  Coalition  Consent  and  the  Change 
in  Iraqi  Government  Consent  from  the  Sunni  Population 

2.  Security 

Security  is  defined  as  the  perceived  risk  of  violent  death  of  a  population  agent.  It 
is  a  function  of  the  level  of  criminality,  civilian  casualties,  and  collateral  damage. 
Looking  at  a  scatter  plot  matrix,  we  see  that  security  across  the  entire  region  of  Iraq  is 
correlated  with  each  individual  region,  and  therefore  we  can  use  the  overall  average  as  a 
good  initial  statistic  for  analysis.  Although  the  overall  average  Iraqi  security  level  is  not 
as  strongly  correlated  as  consent,  it  is  important  to  note  that  this  includes  the  entire 
country  and  all  ethnic  groups.  The  lowest  region  correlation  is  .75,  which  is  still 
significant  enough  to  use  the  average  security  across  Iraq  as  a  response  variable. 
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Correlations 
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Figure  33.  Correlations  of  Security  by  Region  and  the  Overall  Country  of  Iraq 


Histograms  of  the  security  output  show  that  security  after  twelve  months  has  a 
substantial  dispersion.  Security  started  at  5.8  for  this  scenario,  so  it  has  both  increased 
and  decreased  in  this  DOE. 


AVGJraq  Security 


rj-rirr 

r-TL-T 

Tn-h 

-0.3°  | 
-0.20  Q 
-O.IO 

i  i— i  i  i  i 

3  4 

5 

6  7 

Quantiles 

100.0% 

maximum 

7.2022 

99.5% 

7.0124 

97.5% 

6.7751 

90.0% 

6.3209 

75.0% 

quartile 

5.8981 

50.0% 

median 

5.3414 

25.0% 

quartile 

4.6550 

10.0% 

4.1119 

2.5% 

3.6471 

0.5% 

3.4669 

0.0% 

minimum 

3.0728 

Moments 

Mean 

5.279968 

Std  Dev  0.8312745 

Std  Err  Mean  0.0148822 

upper  95%  Mean  5.3091479 
lower  95%  Mean  5.2507881 
N  3120 


Figure  34.  Histogram  and  Quantiles  of  Final  Security  in  Iraq  for  Scenario  DOE 
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Next,  using  a  quadratic  stepwise  linear  regression  allowing  for  interactions,  we 
get  a  telling  model.  Once  again,  we  look  for  the  point  at  which  adding  factors  into  the 
model  yields  minimal  improvement.  This  results  in  an  R-squared  of  0.76,  with  only  nine 
of  the  original  twenty  one  factors  entering  the  model.  Figure  35a  shows  the  summary  of 
fit  and  Figure  35b  is  the  scaled  estimates  description  of  the  model  from  JMP  7.  This 
output  shows  not  only  the  model,  but  nicely  displays  the  scaled  effect  each  regressor  has 
on  the  response  (security).  Bars  that  are  to  the  left  have  a  negative  effect  on  the  response 
and  bars  to  the  right  have  a  positive  effect. 


a. 


Summary  of  Fit 

RSquare 

0.76273 

RSquare  Adj 

0.761353 

Root  Mean  Square  Error 

0.40609 

Mean  of  Response 

5.279968 

Observations  (or  Sum  Wgts) 

3120 

[  Analysis  of  Variance 


Source 

DF 

Sum  of 

Squares 

Mean  Square 

F  Ratio 

Model 

18 

1643.8997 

91.3278 

553.8065 

Error 

3101 

511.3833 

0.1649 

Prob  >  F 

C.  Total 

3119 

2155.2830 

0.0000* 

b. 


Sorted  Parameter  Estimates 

Term 

Coalition  ROE 
Sunni  Marginal  Gains 

Sunni  A  IF  Stance[1]*( Coalition  Stance{2-8&0}+0.25) 

Sunni  ROE 

Sunni  A  IF  Stance[1]*( Coalition  Stance{2&8&0-1}-0.5) 

Coalition  Stance{2-8&0} 

Sunni  AIF  Stance[1] 

Coalition  Stance{2&8&0-1} 

(Coalition  Stance{2-8&0}+0.25)*( Coalition  ROE-3.00615) 

(Coalition  Stance{2&8&0-1}-0.5)*(Sunni  ROE-3.00615) 

(Coalition  Stance{2&8&0-1}-0.5)*( Coalition  Cas  Tolerance-50.0308) 

(Coalition  Stance{2&8&0-1}-0.5)*Coalition  intel  Relationship  w  ith  Sunni  AIF[0] 

Sunni  Force  Protection 

Coalition  Force  Protection 

Coalition  Cas  Tolerance 

Coalition  intel  Relationship  with  Sunni  AIF[0] 

(Coalition  Force  Protection-3.0061 5)*(Sunni  Force  Protection-3.00615) 
(Coalition  ROE-3.0061 5)*(Sunni  Marginal  Gains-0.45003) 


Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

0.2419903 

0.006204 

39.01 

<0001* 

-3.190099 

0.082689 

-38.58 

<0001* 

0.3659304 

0.00958 

38.20 

<0001* 

0.2049923 

0.006204 

33.04 

l 

<0001* 

-0.265439 

0.009534 

-27.84 

1 

<0001* 

-0.250868 

0.009459 

-26.52 

<0001* 

-0.205509 

0.007889 

-26.05 

1  | 

<0001* 

-0.219639 

0.009098 

-24.14 

1 

<0001* 

0.1140548 

0.007482 

15.24 

■ 

<0001* 

0.1025021 

0.007163 

14.31 

i 

<0001* 

-0.003675 

0.000287 

-12.83 

1 

<0001* 

0.1055139 

0.008876 

11.89 

■ 

<0001* 

0.0711882 

0.006204 

11.47 

n 

<0001* 

-0.066782 

0.006205 

-10.76 

1 

<0001* 

0.0019135 

0.000248 

7.71 

j 

<0001* 

-0.041781 

0.007514 

-5.56 

C 

<0001* 

0.0364385 

0.006554 

5.56 

p- 

<0001* 

-0.20074 

0.095331 

-2.11 

s  f  1 

L 

0.0353* 

Figure  35.  Meta  Model  Results  for  Scenario  DOE  in  Regards  to  Security 


Immediately,  the  significance  of  ROE  for  both  the  Sunni  Nationalists  and  the 
coalition  forces  becomes  evident.  Figure  36  shows  that  although  there  are  possible 
exceptions,  we  see  an  increase  in  overall  security  as  both  coalition  and  Sunni  Nationalist 
ROE  increase. 
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Contour  Plot  for  AVGJraq  Security 


AVGJraq  Security 

■  <=  4.500 

■  <=  4.750 

■  <=  5.000 

■  <=  5.250 

■  <=  5.500 

■  <=  5.750 

■  <=  6.000 
■  <=  6.250 

>6.250 


Figure  36.  Contour  Plot  of  Sunni  Nationalist’s  ROE  and  Coalition  ROE  as  a  Function 

of  Average  Security  in  Iraq 


The  next  factor  of  significant  importance  is  that  of  marginal  gains.  What  is  non- 
intuitive  is  that  the  marginal  gain  for  security  is  not  found  in  this.  Looking  into  the 
security  algorithm  we  found  that  this  is  primarily  because  in  this  scenario  we  have  fifteen 
marginal  gains  including  security,  so  security  only  has  one-fifteenth  the  influence  in  the 
population’s  decision  making.  Also,  the  marginal  gains  were  limited  between  .3  and  .6  to 
prevent  extreme  values  of  the  parameter  dominating  the  model. 

Figure  37  displays  this  relationship  through  a  bivariate  fit  of  the  response  security 
in  regards  to  marginal  gains.  There  is  a  clear  underlying  linear  relationship  between 
marginal  gains  and  security.  The  correlation  between  the  two  is  -.34.  This  explains  the 
significance  in  marginal  gains  and  the  importance  of  the  assumptions  which  create  these 
attributes  in  regard  to  the  population.  However,  there  is  still  considerable  variability  in 
this  trend. 
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Figure  37.  Bivariate  Fit  of  Average  Iraq  Security  (Response)  in  Regards  to  Sunni 

Marginal  Gains  (Regressor) 

The  importance  of  stances  for  both  the  coalition  and  the  Sunni  Nationalists  are 
also  evident  in  the  meta  model.  This  finding  corresponds  with  the  third  value  in  Figure 
35  which  explains  our  meta  model.  Equally  interesting  is  that  the  interaction  of  stances  is 
more  significant  than  the  stances  themselves. 

Based  on  the  meta  model,  coalition  stance  is  much  more  significant  toward  the 
nation’s  security  than  toward  consent.  The  plot  of  average  security  levels  by  coalition 
stances  (Figure  38)  provides  insight  into  the  model.  The  coalition  stance  attack  (Level  1) 
provides  the  best  security  on  average  followed  by  humanitarian  aid  (0)  and  withdrawn 
(8),  which  are  not  significantly  different  from  one  another.  Finally,  the  worst  stance  is 
coalition  securing  with  patrols,  which  is  statistically  less  than  the  others.  The  differences 
among  the  means  are  all  statistically  significant  except  for  the  differences  between 
withdrawn  and  humanitarian  aid.  However,  the  highest  values  of  security  are  found 
when  the  coalition  is  withdrawn  or  only  providing  humanitarian  aid.  This  is  a  result  of 
interactions  between  faction  stances. 
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Quantiles 


Level 

Minimum 

10% 

25% 

Median 

75% 

90% 

Maximum 

0 

3.426129 

3.994911 

4.632386 

5.276936 

5.874276 

6.269468 

7.13993 

1 

3.43832 

4.624312 

5.302185 

5.780587 

6.206263 

6.639774 

7.074497 

2 

3.072836 

3.977545 

4.377682 

4.995172 

5.514137 

5.958448 

6.847882 

8 

3.427015 

3.998793 

4.589194 

5.238212 

5.856593 

6.215663 

7.202178 

[  Wilcoxon  /  Kruskal-Wallis  Tests  (Rank  Sums)  | 

(Mean- 

Level 

Count 

Score  Sum 

Score  Mean 

Mean0)/Std0 

0 

780 

1176108 

1507.83 

-1.886 

1 

780 

1584185 

2031.01 

16.844 

2 

780 

948450 

1215.96 

-12.334 

8 

780 

1160017 

1487.20 

-2.624 

1-way  Test,  ChiSquare  Approximation 


ChiSquare  DF  Profc»ChiSq 

334.7291  3  <0001* 


Figure  38.  Means  Diamond  Plot  of  Iraq  Security  in  Regards  to  Coalition  Stance 


Looking  at  just  the  Sunni  AIF  stance  in  Figure  39,  we  see  that  although  the 
difference  is  not  tremendous,  there  is  a  difference  in  security  between  when  the  Sunni 
Nationalists  are  withdrawn  or  present.  The  lack  of  a  tremendous  measured  difference  can 
be  attributed  to  the  actual  game  play  and  is  discussed  in  detail  in  the  conclusion. 
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Quantiles 

Level  Minimum  10%  25%  Median  75%  90%  Maximum 

1  3.25992  4.004505  4.538555  5.147196  5.714113  6.164956  7.074497 

8  3.072836  4.332493  5.161951  5.742247  6.125022  6.559891  7.202178 

Wilcoxon  /  Kruskal-Wallis  Tests  (Rank  Sums) 

(Mean- 

Level  Count  Score  Sum  Score  Mean  Mean0)/Std0 

1  2080  2883766  1386.43  -15.265 

8  1040  1984994  1908.65  15.265 

2-Sample  Test,  Normal  Approximation 
S  Z  Profc»|Z| 

1984994  15.26484  0.0000* 

1-way  Test,  ChiSquare  Approximation 
ChiSquare  DF  Prob>ChiSq 

233.0159  1  <0001* 


Figure  39.  Means  Diamond  Plot  of  Average  Iraq  Security  in  Regards  to  Sunni 

Nationalist  Stance 


Looking  back  at  the  meta  model,  the  interaction  of  stances  is  clearly  significant. 
Figure  40  is  an  interaction  plot  for  the  factors  in  regards  to  the  final  level  of  security.  The 
interaction  plot  provides  two  pieces  of  information.  The  steeper  the  regressors 
corresponding  line,  the  more  significant  the  factor,  and  if  two  regressors  are  not  parallel, 
there  is  likely  an  interaction.  Thus,  when  the  lines  are  crossed  this  alludes  to  strong 
interactions  between  the  regressors.  This  particular  interaction  plot  shows  strong 
interactions  amongst  the  coalition  stance  and  Sunni  Nationalist  stance  factors. 
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The  intent  of  Figure  41  is  to  tease  out  the  significance  of  the  interactions 
discovered  in  our  model  by  a  visual  representation  of  the  actual  data.  The  graph  is 
coalition  stance  within  Sunni  Nationalist  stance  in  regards  to  security.  We  see  that  when 
the  Sunni  Nationalists  are  actively  attacking  (Stance  1)  the  coalition  stances  that  increase 
security  are  1  and  2  (attacking  and  securing  respectively),  which  are  both  offensive 
stances.  When  the  Sunni  Nationalists  withdraw  (Stance  8)  the  coalition  are  best  to  either 
withdraw  or  provide  humanitarian  aid.  We  already  know  that  coalition  stance  is 
significant  in  terms  of  the  mean;  however,  looking  at  the  highest  security  values  per 
stance  in  Figure  40,  we  see  this  as  well. 
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Level  to  Stance 
0  Humanitarian  Aid 

1  Attack 

2  Secure/Patrol 
8  Withdrawn 


Figure  41.  Interactions  of  Coalition  and  Sunni  Nationalist  Stances  in  Regards  to 

Security  [Best  viewed  in  color] 


Also  of  interest  is  the  importance  of  the  interaction  for  coalition  stance  and  ROE. 
Figure  42  is  a  fitted  plot  of  the  security  data  with  coalition  stance  held  constant  at 
securing  (2)  or  attacking  (1).  There  is  a  significant  relationship  which  is  also  intuitive 
between  ROE  and  security.  Tighter  ROE  results  in  higher  security  for  both  offensive 
stances. 
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Stance- 


Stance  - 


Bivariate  Fit  of  AVG lraq  Security  By  Coalition  ROE 


- Linear  Fit 

Secure 


Bivariate  Fit  of  AVG lraq  Security  By  Coalition  ROE 


- Linear  Fit 


Attack 


Figure  42.  Fitted  Plot  of  All  the  Security  Data  with  Coalition  Stance  Held  Constant  at 

Securing  (2)  or  Attacking  (1) 


The  significance  of  force  protection  and  casualty  tolerance  is  also  of  note. 
Although  not  the  most  significant  of  variables,  their  introduction  shows  the  depth  of  the 
security  algorithm  of  PSOM.  Force  protection  determines  the  risk  a  unit  is  willing  to 
take  in  order  to  accomplish  its  mission.  The  meta  model  shows  the  greater  the  force 
protection  level  (which  results  in  more  aggressive  contact)  the  lower  the  security  value. 
However,  casualty  tolerance  has  a  positive  effect  on  security  in  our  model.  Casualty 
tolerance  directly  affects  the  enemy’s  ability  to  deter  the  coalition.  Therefore,  a  high 
casualty  tolerance  will  ensure  the  coalition  continues  its  mission  cycle  in  that  unit’s  area 
of  operations. 

Finally,  it  is  important  to  ensure  that  security  and  consent  are  truly  different 
MOEs,  although  they  should  have  some  sort  of  relationship.  Redundant  MOEs  can  result 
in  unwarranted  reward  for  the  wargame  participants  or  irrelevant  post  analysis. 
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Figure  43  shows  a  correlation  between  the  mean  Iraqi  consent  and  mean  Iraqi 
security.  There  is  a  slightly  positive  correlation  between  the  two  responses,  but 
inarguably  they  are  different  MOEs. 

Correlations 

AVGJraq  Security  Sunni  Consent  toward  Coalition 
A VG_lraq  Security  1.0000  0.1137 

Sunni  Consent  toward  Coalition  0.1137  1.0000 


Figure  43.  Correlation  between  the  Mean  Iraqi  Consent  and  Mean  Iraqi  Security 

C.  SETTINGS  SPECIFIC  DESIGN  OF  EXPERIMENT 

The  settings  file  encompasses  all  the  underlying  assumptions  that  are  not  scenario 
specific,  such  as  the  size  of  an  infantry  company,  or  the  attributes  of  a  particular  stance. 
This  design  of  experiments  consists  of  sixty-six  continuous  factors.  Each  factor  is 
adjusted  to  ±  20%  of  the  initial  value  used  by  DSTL  when  creating  this  particular  settings 
file.  This  prevents  any  one  factor  from  dominating  the  experiment.  When  looking  at  the 
settings  file,  focusing  on  only  the  contributing  variables  allows  this  research  to  explore 
particular  mathematical  models  within  PSOM.  The  primary  goals  in  looking  at  the  game 
settings  are  to  determine  if  PSOM  can  be  used  to  assess  changes  to  unit’s  abilities  and  to 
determine  if  there  are  settings  which,  if  not  truly  understood,  can  drive  the  game. 

1.  Consent 

The  first  MOE  used  as  a  response  is  consent  toward  the  coalition.  Figure  44  is  a 
scatter  plot  matrix  showing  once  again  that  the  overall  average  Sunni  consent  for 
coalition  is  correlated  with  the  consent  in  each  region  and  therefore  is  a  good  metric  to 
use  for  initial  analysis. 
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1.8  2  2.2  1.8  2  2.2  1.9  2.1  2.3  2.1  2.3  1.7  2  2.2 

Region  Region  Region  Region  Region 

Mean  1  Mean  2  Mean  3  Mean  4  Mean  5 


Figure  44.  Scatter  Plot  Matrix  Showing  the  Overall  Average  Sunni  Consent  for 
Coalition  is  Correlated  with  the  Consent  in  Each  Region 

Looking  at  a  histogram  of  the  output,  and  remembering  the  starting  condition  for 
the  scenario  is  the  Sunni’s  having  consent  of  2.5  for  the  coalition,  we  see  that  regardless 
of  settings  consent  does  not  increase.  Also,  the  range  for  consent  change  is  not  very 
substantial  in  regards  to  the  scale  of  possible  outcomes.  It  appears  that  the  overall 
outcome  of  the  simulation  is  not  particularly  sensitive  to  the  experimental  factors. 
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OverallAVGConsent 
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Quantiles 

100.0% 

maximum 

2.3694 

99.5% 

2.3408 

97.5% 

2.3120 

90.0% 

2.2702 

75.0% 

quartile 

2.2284 

50.0% 

median 

2.1639 

25.0% 

quartile 

2.0883 

10.0% 

2.0266 

2.5% 

1.9563 

0.5% 

1.8905 

0.0% 

minimum 

1.8174 

Moments 

Mean 

2.1542624 

Std  Dev 

0.0949538 

Std  Err  Mean 

0.0030027 

upper  95%  Mean 

2.1601548 

low  er  95%  Mean 

2.1483701 

N 

1000 

Figure  45.  Histogram  and  Quantiles  of  Sunni  Average  Consent  toward  Coalition 

from  the  Settings  DOE 


Figure  46  shows  the  results  of  a  stepwise  least  square  regression  meta  model  with 
two-way  interactions.  This  model  provides  an  initial  look  into  the  response  of  mean  Iraq 
consent  for  the  coalition  based  on  the  controlled  settings  factors.  The  most  significant 
factor  in  the  model  is  the  mean  ROE  setting.  This  coincides  with  the  importance  of  ROE 
throughout  the  model.  Although  not  well  defined,  the  author’s  understanding  is  the  mean 
ROE  can  be  related  as  the  expected  ROE  of  a  particular  area  (otherwise,  why  make  it 
variable).  The  recommended  mean  ROE  is  3  (Parkman,  2008);  however,  varying  this 
factor  can  have  a  substantial  effect  on  consent.  Secondly,  the  population  memory 
coefficient  is  a  significant  factor  in  regards  to  consent.  The  greater  this  value,  the  less 
consent  changes  per  time  step.  This  makes  sense;  however,  this  value  is  a  powerful 
assumption  about  the  population. 
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(Adjusted  R  Squared  .937) 


Sorted  Parameter  Estimates 

Term 

Estimate 

t  Ratio 

Prob>|t| 

Mean ROE 

-0.210931 

-97.21 

0.0000* 

PbpulationMemoryCoef 

0.1076177 

49.43 

1 j 

<0001* 

ROEMod 

-0.634123 

-29.22 

[ 

_ 

<0001* 

Force  Protection  Mean 

-0.053347 

-24.57 

T 

<0001* 

UnitProtection 

0.0014413 

22.10 

_ 

i 

<0001* 

UnitFirePower 

-0.001788 

-21.92 

I 

<0001* 

InterUnitBaseCasualtiesDEF 

-1.296548 

-19.87 

1 

□ 

<0001* 

UNitCollateralDamage 

-0.013793 

-14.84 

t 

<0001* 

(PopulationMemoryCoef-3.0006)*(MeanROE-3.0006) 

0.0531651 

8.45 

] 

<0001* 

(UnitFirePOwer-80.016)*(UnitProtection-1 00.02) 

1.5352e-5 

2.16 

r 

0.0309* 

(ForceProtectionMean-3.0006)*(ROEMod-0. 30006) 

-0.024138 

-0.38 

0.7067 

(UNitCollateralDamage-7.0014)*(lnterUnitBaseCasualtiesDEF-0. 10002) 

-0.011726 

-0.14 

0.8851 

Figure  46.  Meta  Model  for  Sunni  Consent  toward  the  Coalition 


Looking  at  a  contour  plot  of  the  two  most  significant  factors  identified  from  the 
meta  model  (Figure  47),  we  see  that  although  other  factors  are  having  some  effect,  these 
primary  two  factors  have  a  tremendous  impact  on  consent.  The  lowest  consent  values  are 
obtained  with  a  low  memory  coefficient  and  a  high  mean  ROE.  The  converse  is  true  as 
well. 
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Contour  Plot  for  OverallAVGConsent 
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Figure  47.  Contour  Plot  Showing  Effects  of  Mean  ROE  and  Population  Memory  on 

Consent  [Best  viewed  in  color] 


A  single  factor  least  squares  regression  provides  a  more  thorough  understanding 
of  the  significance  of  mean  ROE.  Figure  48  shows  we  can  obtain  an  R-squared  of  .59 
with  mean  ROE  as  the  only  regressor.  This  shows  the  exceptional  importance  of  this 
factor. 
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OverallAVGConsent  =  2.7863077  -  0.2106396*MeanRQE 

Summary  of  Fit 

RSquare  0.591114 

RSquare  Adj  0.590704 

Root  Mean  Square  Error  0.060748 

Mean  of  Response  2.154262 

Observations  (or  Sum  Wgts)  1000 

Analysis  of  Variance 

Sum  of 

Source  DF  Squares  Mean  Square  F  Ratio 

Model  1  5.3242804  5.32428  1442.778 

Error  998  3.6829183  0.00369  Prob  >  F 

C.  Total  999  9.0071988  <.0001* 

Parameter  Estimates 

Term  Estimate  Std  Error  t  Ratio  Prob>|t| 

Intercept  2.7863077  0.01675  166.34  0.0000* 

MeanROE  -0.21064  0.005545  -37.98  <.0001* 


Figure  48.  Linear  Regression  Meta  Model  for  Consent  using  only  Mean  ROE  as  a 

Regressor 


The  importance  of  mean  ROE  suggests  that  it  is  worth  looking  into  its  use  in  the 
algorithms  which  determine  civilian  casualties.  The  following  equations  and  their 
explanation  are  taken  from  Paragraph  167  of  the  Peace  Support  Operations  Model 
Specifications  Manual  (Jon  Parkman,  2008).  These  equations  explain  an  important  part 
of  the  combative  algorithms  in  the  simulation.  The  highlighted  equations  take  into 
account  the  mean  ROE  and  the  ROE  modifier  which  is  used  to  control  the  level  which 
rules  of  engagement  actually  impact  civilian  casualties  (Jon  Parkman,  2008). 
Mathematically  p  (mean  ROE)  is  used  in  an  exponential  role  in  determining  the  number 


of  civilians  killed  due  to  combat. 
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Co  =  Rm( 

(e^~Rl.C2] 

!+! 

Where: 

FP  =  Contact  Force  Protection  value  of  Units  1  and  2  (0  <  F  <  1) 

fm  =  Force  protection  modifier  (0  <  fm  <  1) 

fi,  f2  =  Force  protection  values  of  Units  1  and  2  (1  <  f  <  5) 

Pf=  Mean  Force  Protection  value  -  (Typically  3) 

Ci,  C2  =  Casualties  taken  by  Units  1  and  2  (0  <  C  <  00 ) 

BAtt  =  Average  number  of  casualties  taken  in  offensive  operations  of  size  Bc 
(from  HA,  SIGACTS  and  other  sources)  (Constant) 

Boef  =  Average  number  of  casualties  taken  in  defensive  operations  of  size  Bc 
(from  HA,  SIGACTS  and  other  sources)  (Constant) 

Bc  =  Average  size  of  contact  (from  HA,  SIGACTS  and  other  sources)  (Constant) 
Si,  S2  =  Size  of  force  in  contact  from  Units  1  and  2  (0  <  S  <  00 )  -  where  Unit  2 
may  represent  multiple  defending  units. 

Fi,  F2  =  Average  Firepower  values  of  Units  1  and  2  (0  <  F  <  00 ) 

Pi,  P2  =  Average  Protection  values  of  Units  1  and  2  (0  <  P  <  00 ) 

Tf,  Tp  =  Terrain  modifiers  on  Firepower  and  Protection  (0  <  T  <  2) 

I  =  Indirectness  level  of  Unit  1  (0  <  I  <  1) 

Rm  =  RoE  modifier  (0  <  Rm  <  1 ) 

Ri,  R2  =  RoE  levels  of  Units  1  and  2  (1  <  R  <  5) 
pr  =  Mean  RoE  level  -  (typically  3) 

Cav  =  Casualty  level  taken  by  civilians  (0  <  CciV  <  00 ) 

In  a  real  world  situation,  it  makes  sense  that  the  casualty  level  of  civilians  should 
play  a  key  part  in  the  consent  of  a  population.  Furthermore,  the  ability  to  adjust  the 
impact  is  important  in  designing  the  wargame.  This  also  shows  the  importance  of  the 
value  of  security.  Although  it  is  not  directly  involved  in  consent,  because  the  majority  of 
coalition  forces  have  assumed  a  stance  that  involves  providing  security  in  this  scenario, 
the  ways  in  which  the  coalition  affects  the  marginal  gain  of  security  is  key.  In  changing 
the  settings  file  we  have  either  increased  or  decreased  the  number  of  civilians  killed  and 
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therefore  changed  the  civilian  perception  of  security  which  has  an  impact  on  consent. 
This  is  a  good  example  of  the  robustness  of  PSOM  in  regards  to  irregular  warfare 
modeling.  However,  the  mean  ROE  in  this  game  is  very  sensitive  and  can  outweigh  a 
strategic  plan  in  the  game  if  allowed. 

The  memory  coefficient  is  a  scalar  which  determines  the  speed  at  which  a 
population  agent’s  consent  can  change.  The  following  equation,  taken  from  the  PSOM 
manual,  shows  part  of  the  consent  algorithm  and  the  memory  coefficient’s  role: 


Where: 

C'  =  The  final  consent  rating  for  faction  f  in  turn  t 
=  The  consent  rating  for  faction  f  in  turn  t-1. 
kM  =  The  memory  coefficient.  (Jon  Parkman,  2008) 

What  becomes  evident  is  that  the  larger  the  memory  coefficient  is,  the  less 
sensitive  the  change  in  consent  will  be.  This  includes  even  small  changes  such  as  the 
range  covered  in  this  experiment  (2.4  to  3.6).  The  importance  of  this  parameter  makes 
sense  because  of  its  direct  relationship  with  the  final  consent.  It  is  crucial  to  understand 
that  this  value,  if  misused,  can  cause  serious  issues  in  the  simulation.  This  implication  is 
particularly  true  if  model  is  used  as  a  wargame  over  a  short  period  of  time.  If  players  do 
not  see  the  effects  of  their  decisions,  interest  can  be  quickly  lost. 

Looking  back  at  the  meta  model,  the  unit  attributes  of  firepower  and  protection 
are  significant  in  regards  to  consent.  As  unit  firepower  increases  the  consent  goes  down. 
This  is  most  likely  due  to  the  number  of  civilians  killed,  which  links  directly  to  the 
marginal  gain  value  of  security.  As  seen  in  the  previous  equations,  this  is  a  very 
simplistic  look  at  combat  interactions.  Just  because  a  unit  possesses  a  tremendous 
amount  of  firepower  does  not  mean  this  firepower  is  projected,  nor  does  it  mean  this 
firepower  will  result  in  civilian  casualties. 
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As  the  protection  attribute  increases  consent  increases  because  this  allows  for 
more  deliberate  focus  on  the  mission.  The  first  reason  is  the  less  causalities  a  unit  takes, 
the  less  likely  they  are  to  be  deterred  from  the  mission.  More  indirectly,  but  the  above 
equations  also  show,  that  the  greater  the  coalition  protection  the  more  coalition  survive 
resulting  in  fewer  civilian  causalities  because  the  coalition  ROE  is  tight.  The  relationship 
between  consent  and  the  unit’s  collateral  damage  level  is  also  negatively  correlated, 
which  is  intuitive. 

The  limited  effects  of  the  unit  attribute  changes  provide  some  key  insights. 
Further  analysis  must  be  conducted  to  prove  the  hypothesis  that  PSOM  can  be  used  to 
gain  information  on  force  capability  modification.  Arguably,  the  addition  or  removal  of 
20%  of  a  maneuver  company’s  capability  should  have  a  significant  effect. 
Unfortunately,  only  three  of  the  ten  attributes  in  the  experiment  proved  significant  when 
changed  ±20%.  This  is  a  bit  disconcerting  if  the  model  is  intended  to  test  unit 
capabilities. 

2.  Security 

Looking  at  the  data  we  see  that  security  does  increase.  It  starts  at  5.8  throughout 
the  country  in  this  scenario.  Figure  49  is  a  histogram  and  analysis  of  the  security  output 
showing  that  security  increased  in  over  95%  of  the  design  points.  Also,  even  though 
security  is  on  the  same  scale  as  consent  (1  to  10),  the  security  response  variable  covers  a 
much  greater  range  (from  5.7  to  7.7). 
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Distributions 
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Figure  49.  Histogram  and  Analysis  of  the  Security  Output 


Figure  50  is  the  scaled  estimate  results  of  the  least  squares  regression  meta  model 
with  two-way  interactions.  Once  again,  mean  ROE  dominates  followed  by  the  ROE 
mod.  Unit  firepower  and  unit  protection  also  prove  significant. 


Scaled  Estimates 

Continuous  factors  centered  by  mean,  scaled  by  range/2 

Scaled 


Term 

Estimate 

t  Ratio 

Prob>|t| 

Intercept 

6.9389927 

2595.85 

0.0000* 

UnitFireFbwer 

-0.092552 

L 

-19.97 

<0001* 

UnitProtection 

0.1024573 

1 

22.15 

<0001* 

ForceProtectionMean 

-0.115134 

■ 

-24.87 

<0001* 

Mean  ROE 

-0.511041 

-110.37 

0.0000* 

ROEMod 

-0.156403 

I 

-33.78 

<0001* 

InterllnitBaseCasualtiesDEF 

-0.095816 

t 

-20.69 

<0001* 

(UnitFirePower-80.016)*(lnterUnitBaseCasualtiesDEF-0. 10002) 

-0.002674 

-0.33 

0.7407 

(UnitProtection-100.02)*(ForceProtectionMean-3.0006) 

0.0171346 

2.13 

0.0333* 

(MeanROE-3.0006)*(ROEMod-0. 30006) 

-0.019127 

-2.41 

0.0160* 

Figure  50.  Non-ordered  Scaled  Estimates  of  Parameters  of  Settings  File  DOE  in 

Regards  to  Security 
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Figure  51  is  a  contour  plot  of  security  in  regards  to  the  mean  ROE  and  ROE 
modifier.  Mean  ROE  clearly  has  a  tremendous  effect  on  security. 
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Figure  5 1 .  Contour  Plot  of  Mean  ROE  and  ROE  Mod  versus  Security 


Referring  to  the  security  meta  model  in  Figure  49,  and  focusing  on  unit  firepower 
and  unit  protection,  we  see  the  same  results  as  from  earlier  with  consent.  An  increase  in 
a  unit’s  firepower  results  in  a  lower-end  state  security,  and  an  increase  in  protection 
results  in  an  increase  in  security.  This  negative  relationship  between  unit  firepower  and 
the  responses  creates  a  pattern  in  the  model:  the  more  firepower  a  unit  has  the  more  likely 
they  are  to  use  it. 

Because  the  meta  models  for  security  and  consent  had  such  similar  results,  further 
analysis  into  the  relationship  between  security  and  consent  for  this  DOE  proves 
beneficial.  Figure  52  is  a  correlation  scatter  plot  matrix  for  security  and  consent  for  the 
1,000  design  points  in  the  settings  file.  With  a  correlation  of  .9,  clearly  there  is  a  nearly 
linear  relationship  between  security  and  consent  for  this  experimental  design.  This  can 
be  attributed  to  the  primary  factor  driving  change  throughout  the  experiment — mean 
ROE.  Because  most  factors  that  have  effects  on  functions  other  than  the  combative 
algorithms  were  not  significant  with  the  20%  interval  used  in  this  experiment,  the 
combative  functionality  of  PSOM  can  account  for  the  majority  of  the  variation  in  both 
security  and  consent. 
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Multivariate 

Correlations 

lraqAVG_Security  lraqAVG_Consent  (Coalition) 
lraqAVG_Security  1.0000  0.9054 

lraqAVG_Consent  (Coalition)  0.9054  1.0000 


560  rows  not  used  due  to  missing  or  excluded  values  or  frequency  or  weight 
variables  missing,  negative  or  less  than  one. 
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Figure  52.  Scatter  Plot  Correlation  Matrix  of  the  Outputs  Consent  and  Security  in  the 

Settings  DOE 

The  settings  files  contribute  to  the  underlying  assumptions  and  factors  of  PSOM. 
Clearly,  over  sensitivity  of  these  functions  could  prove  problematic  in  the  use  of  the 
model.  Of  the  sixty-six  setting  factors  changed,  only  a  handful  proved  sensitive  to  a  20% 
change  from  the  initial  values.  This  provides  some  insurance  that  for  the  most  part 
questionable  settings  will  not  ruin  the  validity  of  the  game.  This  analysis  has  identified 
those  factors  which  should  be  carefully  manipulated.  However,  the  lack  of  sensitivity  to 
unit  attributes  could  prevent  PSOM  from  becoming  a  tool  used  for  comparative  analysis 
of  unit  equipment  or  force  structure. 
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D.  CUMULATIVE  DOE 


Using  information  gained  over  six  months  of  studying  PSOM  and  all  the  lessons 
learned  from  earlier  analysis,  we  devised  a  cumulative  experiment.  In  this  experiment  we 
focused  on  coalition  stance,  unit  attributes,  ethnic  group  characteristics,  and  time.  It  is 
important  to  remember  this  design  used  a  scenario  in  which  the  Sunni  Nationalists  are 
actively  attacking  the  coalition  and  Iraqi  infrastructure. 

1.  Consent  toward  Coalition 


In  this  case,  consent  outcomes  toward  the  coalition  are  moderate  compared  to  the 
previous  experiments.  Figure  53  also  shows  that  consent  increased  in  approximately 
20%  of  data  points  (initial  value  is  2.5). 
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Figure  53.  Histogram  and  Quantiles  for  the  Cumulative  DOE  in  Regards  to  Consent 

toward  Coalition 
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Figure  54  displays  a  stepwise  quadratic  linear  regression  with  two-way 
interactions  with  a  very  small  mean  square  error  and  an  R-squared  of  0.95. 
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Sorted  Parameter  Estimates 
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0.0029* 

(UnitSize-1 00.061  )*(UnitChangeAttitudePerMan-5.51 51 5) 

2.0286e-5 

7.187e-6 

2.82 

0.0052* 

Coalition  Stance[0] 

-0.001035 

0.000486 

-2.13 

0.0344* 

UnitFirePow  er 

-2.176e-5 

1.634e-5 

-1.33 

0.1841 

UnitChangeAttitudePerMan 

8.8625e-5 

0.000182 

0.49 

0.6271 
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7.068e-6 

1.633e-5 

0.43 

0.6655 

UnitSensors 

-3.864e-7 

1.633e-5 

-0.02 

0.9811 

Figure  54.  Least  Squares  Meta  Model  Results  for  the  Cumulative  DOE  in  Regards  to 

the  Response  Consent 


In  this  experiment  we  intentionally  limited  the  range  of  Sunni  marginal  gain 
security  to  between  .  1  and  .2.  However,  it  remains  a  determining  factor  in  this  model. 

The  next  interesting  finding  is  the  importance  of  the  interaction  of  coalition  stance 
and  time  in  months.  According  to  the  model,  consent  toward  the  coalition  is  greater  if  the 
coalition  is  providing  humanitarian  aid  over  a  12-month  period  than  if  the  collation 
provides  aid  over  a  24-month  period.  This  seems  counterintuitive.  The  increase  in 
production  directly  increases  consent  and  over  twenty-four  months  there  is  more 
production  than  over  twelve  months.  However,  the  lack  of  security  provided  by  this 
tactic  over  the  second  year  seems  to  decrease  consent  toward  the  coalition.  This  shows 
the  dynamics  of  PSOM.  From  this  telling  analysis  it  appears  that  a  faction  cannot  just 
“build”  consent. 

Analyzing  the  importance  of  time  in  regards  to  the  data  from  the  experiment,  we 
see  that,  on  average,  a  12-month  simulation  results  in  a  greater  consent.  Figure  55  (left 
side)  shows  a  means  diamond  plot  of  consent  toward  the  coalition  at  the  end  of  both 
twelve  and  twenty-four  months.  We  can  see  consent  is  greater  at  twelve  months  than  at 
twenty-four.  The  right  side  of  Figure  55  is  a  quantile  plot  where  the  red  line  shows  each 
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data  point  for  the  12-month  design  points,  and  the  blue  line  shows  the  data  point  for  each 
24-month  design  point.  For  all  but  the  99%  quantile  twelve  months  has  a  higher  consent 
toward  coalition. 


Figure  55.  Consent  versus  Time  in  Months  for  the  Cumulative  DOE 


Figure  56  shows  the  interaction  between  coalition  stance  and  the  simulation  time 
in  regards  to  consent.  What  is  noticeable  is  that  when  the  coalition  is  providing 
humanitarian  aid  (Stance  0)  we  see  a  decrease  in  consent  from  a  one  year  simulation  to  a 
two  year  simulation.  However,  there  is  a  slight  increase  in  consent  between  year  one  and 
year  two  when  the  coalition  is  in  a  securing  stance  (Stance  2).  This  supports  the  meta 
model’s  (Figure  54)  findings  that  providing  just  humanitarian  aid  is  not  a  simple  path  to 
increased  consent  in  PSOM. 
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Figure  56.  The  Interaction  between  Time  and  Coalition  Stance  in  Regards  to  Consent 

Also,  we  now  see  the  lack  of  significance  of  the  unit  descriptor  variables  (Figure 
54).  According  to  the  regressor  meta  model  the  unit  attributes  are  not  significant  alone. 
However,  they  are  significant  amongst  themselves  with  interactions.  These  interactions 
seem  intuitive.  For  example,  the  increase  in  unit  size  alone  is  not  significant;  however, 
when  combined  with  increasing  the  unit’s  ability  to  change  perception,  we  see 
significance  in  these  parameters’  interaction.  It  is  important  to  remember  these  variables 
were  changed  with  an  exceptional  range  (±50%  the  original  value),  which  may  not  be 
representative  of  reality. 

2.  Security 

Looking  at  the  average  security  values  for  each  design  point  in  Figure  57  we  see 
that  security  always  increased  throughout  the  country  (the  starting  value  was  5.8). 
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Figure  57.  Cumulative  DOE  Histogram  of  Final  Security  Value 


Figure  58  shows  the  results  of  the  quadratic  least  squares  meta  model  with 
security  as  the  response  variable.  The  factors  we  adjusted  accounted  for  a  very  large 
portion  of  the  variance  within  the  simulation. 
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Sorted  Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 
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Time  in  Months  [12] 

-0.174945 

0.00251 
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<.0001* 

Sunni  Marginal  Gains 

-2.467205 

0.084234 

-29.29 

1 
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<.0001* 

UnitFireRower 
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Figure  58.  Regression  Meta  Model  Results  with  Security  as  the  Response  Variable 
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From  the  parameter  estimates  it  appears  that,  just  as  with  consent,  simulation 
time,  coalition  stance,  and  the  Sunni  marginal  gains  are  significant  factors  within  this 
experiment. 

Figure  59  shows  the  means  diamond  plot  of  security  in  regards  to  time  along  with 
the  normal  quantile  plot  of  both  the  12-month  design  points  and  the  24-month  design 
points.  From  this  picture,  security,  unlike  consent,  seems  to  increase  when  the  scenario  is 
executed  over  two  years. 


Figure  59.  Means  Diamond  and  Normal  Quantile  Plots  of  Security  in  Regards  to 

Time 


Figure  60  shows  the  same  data,  but  introduces  the  interaction  of  unit  stances.  In 
both  cases  where  the  collation  stance  is  either  provide  humanitarian  aid  (0)  or  secure 
through  patrolling  (2)  security  increases  between  the  one-year  design  points  and  the  two- 
year  design  points.  However,  we  see  a  greater  quantity  in  this  increase  when  the 
coalition  is  securing.  This  agrees  with  the  analysis  of  consent.  It  seems  that  although  in 
the  first  year  gains  can  be  made  through  the  use  of  humanitarian  aid  a  faction  must  also 
focus  on  security. 
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Figure  60.  The  Interaction  between  Time  and  Coalition  Stance  in  Regards  to  Security 

Looking  just  at  marginal  gains  in  regards  to  security  (Figure  61)  we  see  that  there 
is  a  clear  trend  that  when  marginal  gains  are  higher  security  is  decreased.  The  space 
where  there  is  no  data  is  a  result  of  the  variables  coalition  stance  and  time.  The  darker 
points  are  the  sixty-six  points  where  the  collation  forces  are  patrolling  for  the  one-year 
time  frame. 


Figure  61.  Bivariate  Fit  of  Security  by  Sunni  Marginal  Gains 
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3. 


Time  Step 


The  final  bit  of  analysis  conducted  was  to  determine  the  stability  of  the  model 
with  respect  to  time  step.  For  this  experiment  we  simulated  the  initial  Iraq  scenario  for 
one  year  of  combat  changing  only  the  time  step  for  each  turn  of  the  game.  We  found 
that,  for  consent,  time  step  can  have  a  substantial  effect  on  the  game  as  shown  below  in 
Table  10. 


Time  Step 

Total  number 

of  steps 

Sunni  Consent 

for  Coalition 

Security 

7  Days=l  time  step 

52 

1.28 

7.13 

30  Days=l  time  step 

12 

2.19 

7.23 

60  days  =  1  time  step 

6 

2.36 

7.17 

Table  10.  Security  and  Consent  Responses  Resulting  from  Changing  only  Time  Step 


E.  SUMMARY 

By  leveraging  intelligent  design  of  experiments  with  powerful  computing  we  have 
been  able  to  obtain  a  very  robust  scope  of  output  from  PSOM.  Using  some  basic  data 
mining  techniques  we  can  gain  some  tremendous  insight  into  the  simulation  model.  A 
synopsis  of  the  results  is  as  follows: 

•  The  methodology  used  within  this  thesis  proved  a  useful  process  to 
analyze  social  models  and  produce  viable  results. 

•  The  initial  assumptions  made  about  the  population  can  be  a  controlling 
factor  of  the  game’s  outcome.  This  is  especially  true  with  respect  to  the 
ethnic  group’s  marginal  gains  and  initial  levels  of  consent  toward  a 
particular  faction. 

•  Although  it  is  difficult  to  increase  consent  in  the  game,  it  is  not 
impossible.  A  careful  balance  of  productivity  and  security  provided 
should  result  in  increased  consent. 
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•  Although  there  is  a  stochastic  aspect  to  PSOM,  it  is  minimal.  It  is  a 
wargame  and  variation  primarily  results  from  the  human-in-the-loop 
aspect  of  the  game. 

•  Time  is  an  important  contributor  to  the  game.  This  applies  to  both  the 
overall  simulated  time  and  time  step  between  turns.  There  is  a  difference 
between  twelve  weeks  of  combat  and  twelve  months,  and  this  is 
represented. 

•  The  total  length  of  represented  time  is  important  within  the  simulation. 
When  the  coalition  stance  was  either  provide  humanitarian  aid  or  secure 
by  patrolling,  security  MOEs  were  varied  at  the  end  of  two  years 
compared  to  the  end  of  one  year. 

•  Stances  and  the  interactions  of  stances  between  factions  have  an  impact  on 
the  game’s  results.  This  is  more  obvious  when  looking  at  security,  but  it  is 
true  in  regards  to  consent  as  well. 

•  In  this  scenario  consent  for  the  Iraqi  government  is  correlated  with  the 
consent  for  the  coalition.  They  are  not  the  same,  however.  This  aspect 
was  investigated  directly  and  the  game  represents  this  well. 

•  Consent  and  security  are  quantitatively  different  MOEs.  They  are  not  just 
scaled  differently;  it  is  possible  to  increase  one  and  decrease  the  other. 

•  Player  set  ROE  and  force  protection  can  have  tremendous  implications  in 
the  scenario.  A  risk  adverse  unit  with  restrictive  ROE  will  fare  better  in  a 
stability  focused  scenario,  especially  in  regards  to  increasing  security. 

•  The  game  is  not  overly  sensitive  to  the  initial  settings  parameters  (these 
are  non-scenario  dependent  assumptions).  However,  this  includes  most 
unit  attributes  (excluding  firepower  and  protection).  These  attributes  have 
marginal  effect  when  changed  20%  and  only  a  small  effect  when  changed 
50%. 

•  One  should  be  very  careful  when  manipulating  the  mean  ROE  value  and 
the  ROE  modifier  as  they  have  a  tremendous  impact  on  the  model. 

•  An  increase  in  a  unit’s  firepower  will  decrease  consent.  Regardless  of  the 
argument  for  or  against  this  assumption,  it  is  imperative  that  unit  attributes 
in  the  settings  support  the  unit’s  actual  capabilities  on  the  ground.  For 
example,  most  armored  units  use  HMMWV’s  in  Iraq.  Therefore, 
firepower  should  be  reduced  accordingly. 

•  Consent  decreased  in  the  two-year  model  compared  to  the  one-year  model 
when  the  coalition  stance  was  set  to  provide  humanitarian  aid,  and 
increased  when  the  coalition  stance  was  set  to  secure  through  patrolling. 
This  shows  that  even  though  production  increased,  the  lack  of  security 
dominated  production  in  determining  consent. 
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•  The  developers’  recommendation  of  a  30-day  time  step  should  be  adhered 
to.  Although  changing  the  length  of  the  time  step  had  minimal  effect  on 
security,  it  did  result  in  different  values  in  the  consent  MOE. 

•  Throughout  this  analysis  many  of  the  results  proved  statistically 
significant;  however,  the  results  might  not  be  deemed  practically 
significant.  A  0.2  change  in  consent  could  easily  be  interpreted  as 
irrelevant.  From  the  author’s  experience  with  PSOM,  this  is  primarily  due 
to  the  normalizing  of  the  final  MOE  values  and  could  therefore  be 
adjusted  to  show  greater  impact  from  the  players’  decisions. 
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VI.  CONCLUSIONS 


A.  RESEARCH  SUMMARY 

This  thesis  set  out  to  develop  and  subsequently  implement  a  methodology  to 
quantifiably  analyze  military  models,  which  claim  to  meet  the  social  implications  of  the 
modem  battlefield.  By  using  a  well-vetted  realistic  scenario,  multiple  design  of 
experiments,  various  data  models,  and  cumulative  statistical  analysis  the  study  focused 
on  addressing  some  fundamental  questions  about  the  popular  simulation  model,  the  Peace 
Support  Operations  Model.  The  final  results  of  these  experiments  and  the  corresponding 
analysis  will  provide  multiple  agencies  within  the  Department  of  Defense  substantial 
insight  into  PSOM  and  a  solid  foundation  for  further  research  of  PMESII  models.  This 
chapter  concludes  the  research  and  analysis  conducted  within  this  study. 

B.  RESEARCH  QUESTIONS 

In  developing  the  purpose  of  this  study  we  identified  four  fundamental  concerns 
in  the  realm  of  PMESII  models,  into  which  PSOM  falls: 

•  Identify  the  factors  which  most  dramatically  influence  PSOM’s  output. 

•  Attempt  to  assess  the  accuracy  of  the  Peace  Support  Operation  Model. 

•  Make  recommendations  toward  the  potential  use  of  the  Peace  Support 
Operation  Model. 

•  Conduct  a  risk  analysis  of  the  Peace  Support  Operation  Model. 

This  chapter  discusses  the  results  of  our  analysis  regarding  each  of  these  initial 
issues.  Many  additional  insights  were  gained  through  this  process  and  the  more 
significant  ones  are  addressed. 

1.  Important  Factors 

Over  100  factors  covering  the  space  of  multiple  factions,  units,  population  agents, 
settings,  and  courses  of  action  were  explicitly  explored  in  this  study.  Of  significant 
importance  are  the  factors  that  the  players  of  the  game  can  control.  These  factors,  as  they 
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should,  have  tremendous  implications  in  the  simulation’s  outcomes.  These  parameters 
include  the  units’  stances,  rules  of  engagement  level,  and  force  protection  level. 
Furthermore,  assumptions  about  the  population  and  the  scenario  have  tremendous 
implications  on  the  responses.  Additional  factors,  which  are  analogous  to  the 
simulation’s  setup  and  execution,  were  identified. 

a.  Player  Controlled  Factors 

Arguably,  the  most  fundamental  decision  a  player  representing  a  faction 
must  make  is  which  stance  to  take.  Therefore,  it  is  critical  that  this  decision  have  an 
effect  on  the  game.  We  found  that  the  faction  stance  selection  has  significant  effects  on 
the  population’s  consent  to  that  faction,  the  consent  of  that  faction’s  allies,  and  the 
security  of  the  country.  In  addition,  the  interaction  of  stances  amongst  different  factions 
proves  to  be  important  as  well.  This  also  proves  true  if  a  faction  is  acting  on  behalf  of 
another  faction.  For  example,  the  coalition  stances  in  this  scenario  often  acted  on  behalf 
of  the  Iraqi  government,  resulting  in  consent  for  both  factions  being  correlated. 

In  regards  to  stance  selection,  the  behavior  of  the  security  MOE  seems 
intuitive.  When  the  Sunni  Nationalists  were  in  an  aggressive  stance,  the  highest  average 
security  was  obtained  when  the  coalition  forces  took  an  aggressive  stance.  However,  if 
the  Sunni  Nationalists  are  withdrawn,  security  was  at  its  highest  when  the  coalition  was 
either  withdrawn  or  providing  humanitarian  aid.  Because  security  is  a  measure  of 
perceived  violent  death,  this  seems  logical.  However,  it  is  surprising  to  learn  that 
regardless  of  Sunni  stance,  the  security  values  are  higher  when  the  coalition  took  an 
attacking  stance  as  opposed  to  a  securing  stance.  Nonetheless,  because  PSOM  is  very 
malleable,  the  stances’  parameters  can  be  changed,  and  we  feel  this  is  a  simple  fix  and 
should  be  looked  into  further  by  the  developers  or  users  of  PSOM. 

With  regard  to  overall  Sunni  consent  toward  the  coalition  and  the  Iraqi 
government,  the  stance  of  the  coalition  also  proved  significant.  On  average,  when  the 
coalition  provides  humanitarian  aid  the  consent  MOE  is  higher.  The  amount  of  the 
consent  values  change  is  also  dependent  on  the  Sunni  Nationalists’  stance,  but  the  trend 
stays  true.  By  focusing  in  on  Baghdad  we  found  that  if  the  Sunni  Nationalists  are 
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aggressively  attacking  the  country’s  infrastructure,  merely  providing  humanitarian  aid 
will  do  very  little  for  consent.  However,  with  a  withdrawn  enemy,  the  coalition’s  aid  is 
uninterrupted  and  consent  increases.  We  also  found  that  consent  is  at  its  lowest  when  the 
coalition  takes  on  an  attacking  stance  versus  humanitarian  aid,  securing,  or  withdrawn.  It 
is  here  that  we  see  the  first  complicated  dynamic  in  contemporary  war;  the  way  to 
increase  consent  alone  seems  intuitive — provide  aid.  This  applies  to  security  as  well:  to 
increase  security,  provide  security.  Yet  to  increase  both  security  and  consent  takes 
substantial  planning  on  the  coalition’s  part.  The  second  dynamic  is  shown  when  the 
scenario  lasts  two  years  as  opposed  to  one  year.  We  found  that  the  positive  benefit  of 
providing  aid  when  the  Sunni  Nationalists  are  in  an  aggressive  stance  drops  off 
tremendously  when  the  scenario  is  extended  to  a  second  year.  It  is  in  this  protracted  war 
that  higher  levels  of  consent  can  be  earned  by  the  coalition  by  providing  increased 
security  through  offensive  stances. 

The  rules  of  engagement  level  and  force  protection  level  the  factions 
choose  prove  very  significant  toward  security,  but  have  minimal  effect  on  consent.  The 
results  for  security  were  intuitive;  the  simple  analysis  is  the  more  restrictive  the 
competing  factions’  ROE  the  higher  security.  The  inverse  proves  true  for  force 
protection;  the  greater  the  risk  the  factions  are  willing  to  take,  the  lower  the  security 
level.  We  also  found  significant  interactions  between  ROE  and  stance.  For  example,  a 
faction  using  an  offensive  stance  with  a  loose  ROE  will  result  in  lower  security. 

b.  Assumptions  about  the  Population 

PSOM  is  a  population  centric  model.  Therefore,  it  should  come  as  no 
surprise  that  the  initial  assumptions  made  about  the  population  have  a  tremendous  impact 
on  the  measures  of  effectiveness.  The  initial  consent  level  an  ethnic  group  has  toward  a 
faction  will  continue  to  impact  the  values  for  consent.  Although  this  is  logical,  it  is  a 
powerful  initial  assumption.  Also,  the  marginal  gains  of  an  ethnic  group  can  strongly 
drive  the  level  of  consent  and  security  in  the  simulation.  These  values  represent  the 
importance  a  population  agent  places  on  particular  goods  to  include  security.  They  have 
extreme  implications  on  the  description  of  an  ethnic  group.  If  these  values  are  entered  at 
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too  high  a  level  it  is  almost  impossible  to  increase  consent  and  security  will  decrease 
significantly.  These  values  must  be  considered  with  the  utmost  care  or  both  security  and 
consent  will  be  adversely  affected.  What  is  of  interest  is  that  the  Sunni  marginal  gains 
for  security  have  the  greatest  effect  on  consent  where  the  Sunni  marginal  gains  for  goods 
have  the  greatest  effect  on  security. 

If  the  simulation  is  intended  for  use  in  any  sort  of  analysis  it  would  be 
imperative  to  execute  the  model  under  multiple  initial  conditions  (Gilbert,  2005).  Unless 
of  course,  the  user  can  find  a  subject  matter  expert,  for  example,  that  knows  the 
importance  of  sanitation  to  a  Sunni. 

c.  Model  Settings 

The  settings  file  focuses  primarily  on  those  parameters  that  are  not 
scenario  specific,  such  as  the  number  of  personnel  in  an  infantry  company  or  modifiers 
which  affect  the  mathematical  models  within  the  simulation.  For  the  most  part,  these 
settings  did  not  prove  overly  sensitive  to  change;  however,  a  few  key  insights  were 
gained. 

The  Average  ROE  is  a  modifier  used  primarily  in  the  civilian  casualty 
determination  algorithm.  The  developers  at  DSTL  state  this  value  is  “normally  3”  (Jon 
Parkman,  2008).  By  varying  this  value  we  see  tremendous  changes  in  both  consent  and 
security.  This  parameter  is  significant  enough  to  completely  change  the  outcomes  of  the 
game  and  should  not  be  adjusted  unless  completely  understood  by  the  user.  The  other 
settings  values  which  were  found  significant  are  a  few  basic  modifiers,  primarily  the 
population’s  memory  coefficient  and  the  ROE  modifier  (see  Chapter  IV).  For  the  most 
part  their  manipulation  is  of  minimal  effect  or  easily  understood. 

However,  there  seems  to  be  a  lack  of  effect  pertaining  to  the  changing  of 
unit  capabilities.  We  found  that  changing  a  unit’s  (company  level)  attributes  by  20%  has 
minimal  effect  on  both  security  and  consent.  We  then  changed  these  values  by  over  50% 
from  the  initial  values  and  saw  that  large  changes  in  attributes  can  affect  the  model. 
These  attributes  include  unit  firepower,  sensors,  ability  to  change  populace  attitude,  and 
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unit  size.  The  ability  to  show  that  these  factors  matter  should  assure  the  user  that  unit 
capability  is  important;  however,  the  lack  of  sensitivity  says  move  cautiously  if 
considering  the  use  of  PSOM  as  a  force  analysis  tool. 

The  exception  to  this  is  the  unit’s  firepower  setting  and  protection  level. 
The  protection  level  acts  in  an  understandable  way  by  having  positive  changes  on 
security  and  consent;  however,  firepower  has  the  opposite  effect  on  security  and  consent. 
When  using  the  preset  units  in  PSOM,  it  is  important  to  remember  that  a  unit  with 
tremendous  firepower,  such  as  an  armored  battalion,  might  use  different  equipment  based 
on  the  combat  situation. 

d.  Time  Step  and  Time  of  Simulation 

There  are  two  ways  to  look  at  time  step  and  both  were  analyzed  in  this 
thesis.  First,  it  is  clear  that  the  length  of  the  time  step  is  significant.  As  would  be 
expected,  the  results  of  twelve  7-day  time  steps  provided  significantly  different  results 
than  twelve  30-day  time  steps.  However,  we  found  that  fifty-two  7-day  time  steps 
provided  substantially  different  results  than  twelve  30-day  time  steps  when  looking  at  the 
consent  metric.  Therefore,  it  is  best  to  stay  with  the  developer  recommended  30-day  time 
step. 

Intuitively,  the  results  of  twelve  months  of  game  play  differ  from  those  of 
twenty-four  months  of  game  play  with  regard  to  both  security  and  consent.  Using  the 
standard  Iraq  scenario,  which  portrays  an  aggressive  Sunni  Nationalist  Faction,  we  see 
that  on  average  security  increased  from  12  to  24  months.  However,  when  the  coalition 
chooses  a  stance  of  humanitarian  aid,  there  is  a  decrease  in  consent  from  12  to  24  months. 
Consent  increases  on  average  between  12  and  24  months  provided  the  coalition  chooses  a 
securing  stance  against  the  active  insurgency.  This  implies  that  PSOM  is  much  more 
than  a  simple  model  portraying  linear  relationships  amongst  user  inputs.  The 
implications  of  non-linear  (the  curve  looks  quadratic)  changes  in  consent  over  time  is 
most  likely  the  result  of  an  initial  consent  for  a  coalition  that  provides  life  improving 
services  upfront;  however,  as  the  promise  of  a  better  country  is  not  met  due  to  a  lack  of 
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security  the  consent  decreases.  Where-as  a  coalition  that  initially  provides  marginal 
goods  and  services,  but  provides  and  then  increases  security  can  gradually  continue  to 
gain  consent  from  the  population. 

2.  Accuracy  of  the  Peace  Support  Operation  Model 

Because  social  modeling  is  far  more  complex  than  physics-based  modeling  there 
is  no  binary  answer  as  to  the  accuracy  of  the  model’s  outputs.  However,  through  the 
comparison  of  our  quantitative  results  with  doctrine,  personal  experience,  and  the  help  of 
subject  matter  experts,  subjective  analysis  can  be  obtained.  First  and  foremost,  the  model 
uses  a  multitude  of  player  and  subject  matter  expert  inputs  to  create  a  simulated 
environment  which  describes  irregular  warfare.  In  the  process  of  conducting  this 
research  we  have  explored  a  vast  amount  of  the  PSOM  parameter  space  and  found  many 
doctrinally  analogous  results.  It  is  assuring  that  the  model  is  more  than  an  empty  black 
box  that  provides  output  regardless  of  input.  Also  of  note,  the  model  is  very  flexible. 
There  are  modifiers  that  can  influence  the  importance  of  the  majority  of  the  algorithms 
within  the  simulation.  This  allows  the  user  to  define  the  space  in  which  the  wargame  is 
played  and  therefore  influence  its  accuracy. 

Looking  at  the  consent  MOE  we  have  learned  that  it  is  very  dependent  on  the 
initial  assumptions  about  the  population.  We  have  also  learned  that  it  is  very  difficult  to 
increase  this  factor.  Arguably  this  is  logical,  as  it  is  very  difficult  to  change  the  initial 
perceptions  of  a  population.  However,  what  the  game  does  show,  which  proves  a  level  of 
accuracy,  is  that  a  faction  cannot  just  “build”  consent.  Using  the  base  case  Iraq  scenario 
with  an  active  insurgency,  we  manipulated  the  coalition  combat  units  to  all  be  able  to 
provide  humanitarian  aid  in  a  manner  equivalent  to  humanitarian  aid  organizations.  Over 
a  two-year  simulation  with  all  coalition  units  in  Sunni  populated  areas  (37  battalions) 
focused  on  providing  an  exaggerated  amount  of  aid,  there  is  a  clear  increase  in 
production.  However,  even  under  this  design,  consent  either  stayed  at  the  initial 
conditions  or  eventually  decreased.  However,  with  the  same  units  now  providing 
security,  and  only  non-maneuver  units  providing  aid,  we  see  some  excursions  where 
consent  increased  and  a  slightly  higher  value  in  consent  than  the  aforementioned 


110 


scenario.  These  results  are  encouraging  in  that  you  cannot  just  buy  a  population’s 
consent.  This  falls  into  line  with  the  importance  of  Unity  of  Effort  in  Counterinsurgency 
Operations  as  discussed  in  FM  3-24  (U.S.  Army,  2006). 

Military  efforts  are  necessary  and  important  to  counterinsurgency  (COIN) 
efforts,  but  they  are  only  effective  when  integrated  into  a  comprehensive 
strategy  employing  all  instruments  of  national  power.  A  successful  COIN 
operation  meets  the  contested  population’s  needs  to  the  extent  needed  to 
win  popular  support  while  protecting  the  population  from  the  insurgents. 

This  goes  back  to  the  complexity  of  the  definition  of  consent.  However,  along  the 
same  lines  a  large  part  of  consent  is  production  and  an  important  part  of  production  is 
determined  by  the  ability  to  hire  human  capital.  The  ability  to  hire  is  determined  on  price 
for  labor  and  consent  toward  the  hiring  faction.  This  price  for  labor  is  set,  and  no  faction 
can  change  this  price.  This  places  a  great  restriction  on  wealthy  factions,  such  as  the 
coalition  in  Iraq,  who  can  raise  the  price  paid  for  labor  to  ensure  human  capital  is 
obtained  where  needed. 

Another  capability  PSOM  provides  is  the  ability  to  share  the  credit  for  production 
of  goods  or  the  provision  of  security.  Throughout  this  research  the  coalition  shared  its 
production  credit  with  the  Iraqi  government.  This  did  in  fact  influence  the  population’s 
consent  toward  the  government  which  is  a  key  factor  in  the  development  of  legitimacy 
toward  the  host  nation  (U.S.  Army,  2006). 

In  looking  at  security  we  found  some  effects  that  should  be  explored  further. 
Security  is  greatly  influenced  by  the  unit’s  rules  of  engagement  and  risk  tolerance,  which 
makes  sense.  However,  the  stances,  although  statistically  significant,  are  arguably  not 
significant  enough.  Once  again,  looking  at  two  years  of  combat  in  the  base  case  Iraq 
scenario  where  there  is  an  active  insurgency,  we  found  there  to  be  only  a  slight  difference 
between  all  coalition  maneuver  battalions  providing  aid  and  all  coalition  maneuver 
battalions  securing  through  patrols.  The  mean  difference  between  the  two  is  0.1,  and 
when  looking  at  the  extremes  in  the  final  DOE  where  the  coalition  did  its  best  for  security 
while  patrolling  and  the  coalition  did  its  worst  for  security  while  providing  aid,  the 
difference  is  only  0.5.  Since  security  falls  within  a  range  of  0  to  10,  a  difference  of  0.5, 
while  “statistically  significant,”  may  not  appear  exceptionally  significant  for  the  purpose 
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of  analysis.  The  abilities  of  stances  to  affect  MOEs  can  be  adjusted  within  the  settings; 
however,  this  is  not  a  task  to  be  taken  on  by  an  individual  who  does  not  completely 
understand  the  model’s  algorithms. 

Finally,  as  discussed  earlier,  there  are  questionable  implications  about  the 
absolute  changes  in  unit  characteristics.  Initially  this  seems  illogical;  if  you  make  every 
company  in  Iraq  20%  better  you  would  expect  different  results  in  security  and  consent. 
This  can  be  argued  with  the  recent  surge  of  forces  in  Iraq.  However,  the  recent  success  in 
Iraq  is  much  more  than  a  result  of  increased  manpower  and  capability;  it  is  the  result  of 
the  well-planned  use  of  these  increased  attributes.  It  is  beyond  the  scope  of  this  study 
and  our  current  abilities  to  develop  multiple  courses  of  action  to  test  this  capability  within 
the  model.  This  should  be  studied  prior  to  using  PSOM  as  a  tool  for  capability  analysis. 

3.  Potential  Uses  for  the  Peace  Support  Operations  Model 

PSOM  is  a  campaign  level  wargame  and  therein  lies  its  greatest  potential. 
Combat  is  stochastic  in  nature  (Lucas,  2000),  and  the  addition  of  a  complex  populace, 
IGOs,  and  political  players  only  add  to  the  uncertainty  of  war.  The  limited  stochastic 
nature  of  PSOM  places  a  tremendous  limit  on  using  a  batch  mode  to  analyze  scenarios. 
In  2006  the  Al-Askari  Mosque  in  Samarra,  Iraq  was  destroyed  by  Sunni  insurgents 
resulting  in  violence  across  Iraq.  This  level  of  resolution  would  have  to  be  deliberately 
built  into  the  scenario  file  in  PSOM  and  would  thus  be  limited  to  the  scenario  developer’s 
creativity.  However,  such  realistic  actions  which  can  change  the  state  of  a  campaign  can 
be  expected  to  occur  with  the  human-in-the-loop.  Without  the  human  players  such 
extreme  actions  would  more  than  likely  not  happen.  As  Clausewitz  stated: 

They  aim  at  fixed  values;  but  everything  in  war  is  uncertain,  and 

calculations  have  to  be  made  with  variable  quantities 

— Carl  Von  Clausewitz,  On  War 

The  Peace  Support  Operations  Model  provides  an  opportunity  for  a  large  number 
of  players  to  interact  and  to  learn  from  a  simulated  population.  This  population  is 
dynamic,  consisting  of  the  emersion  of  many  diverse  social  groups  which  have  the  ability 
to  change  over  time.  The  results  of  the  algorithms,  which  are  constantly  being  updated 
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and  are  easily  modified,  have  proven  feasible  within  this  study.  There  is  no  doubt 
valuable  lessons  can  be  learned  for  use  in  staff  development  and  training. 

In  the  same  argument,  wargaming  is  of  tremendous  importance  in  course  of  action 
development.  If  the  manpower  is  available  to  provide  a  human-in-the-loop  wargame, 
PSOM  can  provide  insight  into  stability  operations  where  traditional  lethality  focused 
models  cannot.  Once  a  scenario  is  developed  and  a  plan  implemented,  it  takes  minimal 
effort  to  change  the  underlying  assumptions  about  the  population.  For  example,  a  plan 
can  be  implemented  against  a  multitude  of  modified  populations  and  results  will  vary 
accordingly.  As  long  as  the  decision  makers  understand  that  the  complexity  of  social 
modeling  ensures  no  model  is  a  crystal  ball,  this  implementation  shows  potential. 

As  a  wargame  PSOM  provides  an  environment  that  brigade  and  higher  staffs  can 
easily  work  within.  At  the  same  time  PSOM  provides  a  large  list  of  outputs  which  can 
easily  be  transformed  into  strategic  level  MOEs  to  parallel  current  doctrine.  These 
attributes  strengthen  the  argument  for  its  use  as  a  wargame. 

The  limited  sensitivity  we  found  in  the  model  toward  unit  attributes  causes  some 
initial  concern  about  the  model’s  use  in  force  development.  Based  on  our  results  alone 
the  model  does  not  appear  sensitive  enough  to  attribute  changes.  However,  as  stated 
earlier,  these  forces  were  not  used  in  any  intelligent  manner,  so  the  results  are 
questionable.  Further  analysis  on  this  aspect  of  the  model  is  recommended  prior  to  its 
use  in  force  capability  analysis. 

Also,  PSOM  proves  sensitive  to  time  length  and  therefore  time  step.  Although  it 
is  a  Peace  Support  simulation,  its  use  for  short-term  operations  such  as  disaster  relief  or 
hasty  interventions  could  be  limited.  If  the  needed  response  resolution  from  an  operation 
is  to  be  measured  in  days  or  weeks,  PSOM  could  provide  skewed  results.  The  same 
assumption  can  be  made  for  tactical  level  operations.  The  creators  of  PSOM  recommend 
it  for  task  force  level  use  and  higher.  The  simulation’s  unit  interaction  algorithms  are 
conducted  at  a  company  level  resolution.  Therefore,  further  analysis  is  needed  to 
determine  if  PSOM  could  be  an  effective  tool  for  company  and  below  level  training  and 
course  of  action  development. 
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4. 


Risk  Analysis 


As  with  all  tools  in  the  military,  PSOM  has  inherent  risk  if  misused.  PSOM  will 
not  provide  the  right  answer.  It  will  only  provide  insights  as  to  what  happens  in  its 
simulated  space.  And,  because  it  is  minimally  stochastic,  it  does  not  provide  a  wide 
range  of  possible  outcomes.  This  is  why  the  human-in-the-loop  aspect  is  essential.  Also, 
there  are  a  tremendous  number  of  parameters  which  can  affect  the  game.  This  aspect 
makes  PSOM  very  flexible;  however,  if  these  parameters  are  not  placed  by  an  expert  the 
resulting  MOEs  are  questionable  at  best.  It  is  absolutely  crucial  that  a  scenario  builder 
not  only  know  how  to  change  the  parameters,  but  understand  the  algorithms  he  or  she  is 
changing.  Currently  there  are  draft  manuals  for  PSOM  available  which  need  to  be 
completed  prior  to  its  mass  use  (or  the  scenario  builder  needs  to  have  the  developers  on 
speed  dial). 

Finally,  prior  to  the  use  of  PSOM  as  a  learning  tool  or  for  course  of  action 
development,  the  scenario  and  its  settings  should  be  data  farmed  using  a  similar,  but  less 
extensive,  methodology  to  that  discussed  in  this  thesis.  This  will  verify  the  scenario  and 
help  prevent  questionable  outcomes  from  the  realm  of  possibilities  creative  players  bring 
to  the  wargame.  As  a  participant  in  wargaming,  both  as  a  planner  and  as  a  student, 
nothing  will  shut  down  the  opinion  of  a  military  officer  faster  than  infeasible  results.  The 
data  farming  process  will  act  as  a  large-scale  rehearsal  to  ensure  the  model  and  its 
scenario  are  acceptable  prior  to  gathering  the  expensive  amount  of  manpower  required 
for  such  a  wargame. 

5.  Methodology 

Many  of  the  readers  of  this  document  are  curious  about  the  potential  of  PSOM. 

However,  the  analysis  of  PSOM  was  partially  used  to  verify  the  data  farming 

methodology  as  a  suitable  technique  to  quantifiably  assess  military  simulation  models 

which  account  for  societal  phenomenon.  Since  data  farming’s  introduction  during 

Project  Albert  in  1999  it  has  been  used  repeatedly  on  a  host  of  agent-based  simulation 

packages  to  gain  insight  into  both  the  application  and  the  outcomes.  Dr.  Home  suggests 

its  implementation  in  the  verification  and  validation  process  (Horne,  2004)  and  we  have 
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implemented  it  in  this  thesis  with  success.  By  indentifying  key  factors,  developing 
efficient  experimental  designs,  using  high  performance  computing  power  to  conduct  the 
experiments,  and  analyzing  simulation  output  with  data  mining  techniques  a  wealth  of 
information  can  be  gained  from  a  simulation  that  was  designed  to  be  played  one  step  at  a 
time  over  the  course  of  days.  To  explore  even  a  fraction  of  the  simulated  space  covered 
over  the  past  six  months  would  be  inconceivable  if  it  were  to  be  done  via  a  human-in-the- 
loop  game  play.  Additionally,  this  was  all  accomplished  with  minimal  manpower.  This 
methodology  is  scientific  in  its  background  and  can  provide  substantial  insights  into  the 
rapidly  growing  field  of  PMESII  models. 
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APPENDIX  A.  SCENARIO  EXPERIMENTAL  DESIGN 


The  scenario  design  of  experiment  consists  of  twenty-one  factors.  The  following 
table  shows  the  first  five  design  points  for  the  DOE  and  all  dependent  settings  used.  For 
example,  although  stance  is  an  experimental  factor,  the  substances  must  be  changed  to 
align  with  the  actual  stances  within  PSOM.  Factors  changed  throughout  the  experiment 


are  highlighted. 


DP  1 

DP  2 

DP  3 

DP  4 

DP  5 

NGO  Stance 

8 

8 

8 

8 

8 

NGO  Substance 

4 

4 

4 

4 

4 

Sunni  AIF  Stance 

1 

1 

1 

1 

1 

Sunni  AIF  Sub  Stance 

1 

1 

1 

1 

1 

Sunni  AIF  ATKUS 

0 

0 

0 

0 

0 

Sunni  AIF  ATK  Iraqi  Gov’t 

1 

1 

1 

1 

1 

Coalition  Stance 

8 

8 

8 

8 

8 

Coalition  Sub  Stance 

4 

4 

4 

4 

4 

Coalition  intel  Relationship  with  Sunni  AIF 

0 

0 

0 

0 

0 

Coalition  ROE 

4 

5 

5 

4 

5 

Coalition  Force  Protection 

1 

4 

2 

5 

3 

Sunni  ROE 

2 

1 

5 

4 

2 

Sunni  Force  Protection 

2 

3 

2 

3 

1 

Sunni  Political  Ideology 

13 

34 

30 

6 

9 

Sunni  MG  Security 

0.53 

0.375 

0.558 

0.427 

0.347 

Initial  Threat 

1.2 

0.2 

-1.4 

-0.9 

-1 

Initial  Consent 

-0.1 

1 

-0.2 

-0.9 

0.4 

Normalized  Initial  Consent 

4.750208 

7.310586 

4.50166 

2.890505 

5.986877 

Sunni  Marginal  Gains 

0.591 

0.516 

0.422 

0.338 

0.483 

Coalition  Leadership 

72 

92 

58 

98 

48 

Coalition  Experience 

55 

77 

80 

91 

6 

Coalition  Reputation 

94 

48 

88 

63 

84 

Coalition  Cas  Tolerance 

19 

23 

47 

8 

64 

Coalition  Turns  at  location 

8 

4 

6 

2 

7 

HumanAidF  actionT  argetsandGoodDelivered 

0 

0 

0 

0 

0 
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APPENDIX  B.  SETTINGS  EXPERIMENTAL  DESIGN 


The  settings  design  of  experiment  consists  of  sixty-six  factors.  The  DOE  shown 


below  is  the  first  five  design  points  with  the  inclusive  factor  values. 


DP  1 

DP  2 

DP  3 

DP  4 

DP  5 

UnitFirePower 

93.504 

72.352 

68.864 

77.6 

86.304 

UnitProtection 

99.92 

107.56 

94.52 

96.12 

90.48 

UnitSensor 

59.76 

58.98 

40.4 

53.38 

55.9 

UNitlntel 

22.328 

20.272 

22.68 

17.448 

19.632 

UnitPhysicalCamo 

1.1608 

1.1068 

0.9076 

0.948 

1.186 

UnitSocialCamo 

0.8228 

0.928 

1.0468 

0.8708 

0.9668 

UnitMobilityPerMan 

35.352 

28.236 

29.868 

25.788 

26.88 

UnitChangeAttitudePerMan 

3.2628 

3.2544 

2.7096 

3.4896 

3.3936 

UnitCrime 

1.7784 

2.3424 

1.6608 

2.3856 

2.1088 

UnitPolicing 

2.9292 

3.0072 

2.9364 

2.7132 

2.6496 

UNitCollateralDamage 

7.7588 

7.63 

6.916 

7.6272 

7.1932 

UNitSize 

81 

102 

98 

84 

77 

UnitPalitive 

0.2729 

0.3341 

0.3144 

0.2506 

0.3366 

StanceAttacksUnit 

0.2963 

0.3251 

0.2668 

0.3011 

0.3182 

StanceProtectPopulation 

0.4459 

0.3736 

0.4258 

0.3982 

0.3394 

StanceProvideAid 

0.1688 

0.1663 

0.173 

0.191 

0.189 

StanceModifyPerception 

0.4624 

0.3634 

0.4694 

0.468 

0.4302 

Stance  Extort 

0.2715 

0.0747 

0.2289 

0.1206 

0.1293 

StanceCounterCrime 

0.2555 

0.2863 

0.3086 

0.3084 

0.2899 

StanceINtelGather 

0.5796 

0.446 

0.5658 

0.5134 

0.4982 

StanceQRF 

0.3246 

0.3494 

0.3755 

0.3341 

0.4696 

StanceAVGSize 

30.828 

32.232 

33.204 

26.04 

27.084 

StanceProtection  Modifier 

0.9491 

0.9264 

0.7558 

0.8832 

0.7763 

StanceMobilityModifier 

0.9764 

1.0988 

0.8124 

1.1416 

0.9984 

StanceDetectabilityModifier 

1.415 

1.4141 

1.128 

1.1208 

0.9826 

StanceDetection  Modifier 

1.4952 

1.4526 

1.6902 

1.4214 

1.7004 

StanceRecognition  Modifier 

1.1948 

0.9812 

0.8716 

0.8448 

1.09 

Population  Decision  Radius 

56.28 

48.18 

48.52 

54.84 

49.92 

Population  MemoryCoef 

3.546 

2.556 

2.9712 

3.4176 

3.504 

PopulationConsentPoliticalMScaler 

-0.6954 

-0.9434 

-0.9565 

-0.865 

-0.6634 

PopulationConsentPoliticalCScaler 

0.0855 

0.0807 

0.0933 

0.0979 

0.1185 

PopulationAverageTerm In  Prison 

2.802 

2.538 

2.76 

2.8464 

2.5884 

PopulationSelfPresenters 

0.5826 

0.5448 

0.4596 

0.5152 

0.4714 
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PopulationPoliceClearRate 

0.2518 

0.2385 

0.2919 

0.2268 

0.2923 

Population  Infection  MargGains 

0.2295 

0.2017 

0.2108 

0.1946 

0.2072 

CombatModQueDecayRate 

0.0143 

0.0136 

0.0147 

0.0139 

0.0156 

PercentForceonDuty 

0.2712 

0.3176 

0.2711 

0.3196 

0.3354 

PlanningDelay 

6.5352 

5.9184 

6.132 

4.9296 

5.8296 

OperationTime 

4.6368 

4.1648 

4.3952 

3.8208 

4.3856 

RecupTIme 

10.716 

8.98 

11 

8.62 

11.052 

AvgDistanceTraveled 

7.3024 

8.3008 

8.4256 

7.1136 

6.432 

MaxFatigue 

0.7462 

0.6826 

0.8029 

0.7197 

0.9357 

FatigueDropOff  Factor 

3.6544 

4.3552 

3.712 

4.512 

3.4192 

ForceProtectionMean 

2.6016 

2.6292 

3.0324 

3.1896 

2.4348 

ForceProtectionKValue 

0.3547 

0.2478 

0.3568 

0.3288 

0.2521 

ForceProtectionMod 

0.33 

0.2456 

0.3475 

0.3516 

0.3449 

MeanROE 

2.8176 

2.9484 

3.5784 

2.5032 

3.2736 

ROEKValue 

0.2874 

0.3589 

0.261 

0.2636 

0.2696 

ROEMod 

0.2778 

0.2612 

0.2896 

0.3532 

0.3272 

MaxLeadershipMod 

2.1992 

1.972 

1.612 

1.632 

1.6984 

MinLeadershipMod 

0.1615 

0.1602 

0.2021 

0.1613 

0.2214 

LeadershipDropOff  Factor 

0.0186 

0.0175 

0.0192 

0.0207 

0.018 

FamiliarizationStranger 

0.5306 

0.56 

0.5136 

0.4022 

0.5668 

FamiliarizationNative 

1.8528 

2.1352 

2.1632 

2.1408 

1.9304 

Familiarization  Learniing 

0.0982 

0.0904 

0.1146 

0.105 

0.101 

ExperienceConscript 

0.2037 

0.2291 

0.1912 

0.2024 

0.1704 

ExperienceVEt 

2.244 

1.8736 

1.9704 

1.988 

2.2392 

ExperienceLearningFactor 

0.0842 

0.0965 

0.1047 

0.0879 

0.0752 

InterUnitBaseCasATT 

0.0872 

0.1044 

0.0875 

0.1007 

0.1036 

InterUnitBaseCasualtiesDEF 

0.1082 

0.1086 

0.0905 

0.1044 

0.0852 

InterUNitBaseContactSIze 

29.292 

26.076 

26.52 

29.844 

24.348 

Good  Expected 

0.4174 

0.4004 

0.593 

0.5432 

0.5496 

GoodProtectionValuePower 

10880 

8636 

10856 

11092 

11624 

GoodProtectionValueWater 

64.344 

77.812 

56.364 

83.58 

74.732 

GoodProtectionValueEducation 

63.364 

76.02 

71.54 

65.184 

83.664 

GoodProtectionValueHealthCare 

132.84 

144.54 

166.08 

124.56 

172.62 
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APPENDIX  C.  CUMULATIVE  EXPERIMENTAL  DESIGN 


The  first  five  design  points  of  the  cumulative  experimental  design  are  shown 

below. 


DP  1 

DP  2 

DP  3 

DP  4 

DP  5 

Coalition  Stance 

0 

0 

0 

0 

0 

Simulation  Time 

60 

60 

60 

60 

60 

Unit  Fire  Power 

81 

56 

63 

69 

125 

Unit  Protection 

150 

75 

94 

113 

144 

Unit  Sensors 

131 

138 

56 

81 

94 

Unit  Change  Attitude 

4 

6 

3 

10 

2 

Unit  Manpower 

75 

50 

113 

106 

81 

Unit  Intelligence 

48 

29 

44 

24 

20 
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APPENDIX  D.  CORRELATION  AND  SPACE  FILLING  QUALITY 

OF  FACTORS 


This  appendix  shows  the  pair-wise  correlations  and  space  filling  qualities  for  the 
quantitative  variables  for  each  DOE.  The  use  of  NOLHs  results  in  low  pair-wise 
correlations  while  maintaining  good  space  filling  properties. 


A.  SCENARIO  EXPERIMENTAL  DESIGN 


Correlations 

Coaliion  ROECoaliion  Force  Protection  Sunni  ROE  Sunni  Force  Protection  Sunni  Poliical  Ideology  Sunni  MG  Security  Initial  Threatlniial  ConsentSunni  Marginal  Gains  Coalition  Leadership  Coalition  Experience  Coalition  Reputation  Coalition  Cas  Tolerance  Coalition  Turns  at  location 


Coaliion  ROE 

1.0000 

■0.0000 

■0.0000 

0.0001 

-0.0091 

0.0012 

•0.0006 

0.0074 

0.0054 

•0.0056 

0.0065 

0.0076 

•0.0003 

•0.0015 

Coaliion  Force  Protection 

■0.0000 

1.0000 

•0.0000 

■0.0000 

■0.0032 

0.0068 

•0.0047 

0.0036 

•0.0148 

-0.0129 

•0.0064 

0.0057 

•0.0038 

■0.0179 

Sunni  ROE 

■0.0000 

■0.0000 

1.0000 

■0.0000 

0.0028 

0.0193 

-0.0051 

0.0025 

-0.0024 

0.0037 

■0.0012 

■0.0118 

•0.0032 

•0.0130 

Sunni  Force  Protection 

0.0001 

■0.0000 

•0.0000 

1.0000 

■0.0019 

-0.0033 

0.0069 

0.0039 

0.0068 

0.0064 

•0.0014 

•0.0092 

■0.0042 

•0.0008 

Sunni  Poliical  Ideology 

-0.0091 

■0.0032 

0.0028 

■0.0019 

1.0000 

-0.0017 

-0.0045 

■0.0037 

-0.0008 

-0.0042 

•0.0044 

■0.0056 

•0.0000 

0.0107 

Sunni  MG  Securiy 

0.0012 

0.0068 

0.0193 

■0.0033 

■0.0017 

1.0000 

•0.0095 

0.0017 

•0.0010 

•0.0019 

0.0091 

•0.0122 

0.0070 

■0.0177 

Iniial  Threat 

■0.0006 

■0.0047 

-0.0051 

0.0069 

■0.0045 

■0.0095 

1.0000 

■0.0033 

-0.0201 

-0.0064 

•0.0069 

0.0049 

0.0039 

0.0048 

Iniial  Consent 

0.0074 

0.0036 

0.0025 

0.0039 

■0.0037 

0.0017 

•0.0033 

1.0000 

■0.0086 

0.0147 

•0.0005 

•0.0151 

0.0083 

0.0026 

Sunni  Marginal  Gains 

0.0054 

■0.0148 

•0.0024 

0.0068 

-0.0008 

-0.0010 

-0.0201 

•0.0086 

1.0000 

0.0163 

•0.0037 

■0.0075 

•0.0182 

•0.0035 

Coaliion  Leadership 

■0.0056 

■0.0129 

0.0037 

0.0064 

■0.0042 

•0.0019 

•0.0064 

0.0147 

0.0163 

1.0000 

0.0067 

0.0098 

0.0039 

•0.0244 

Coaliion  Experience 

0.0065 

•0.0064 

•0.0012 

■0.0014 

•0.0044 

0.0091 

•0.0069 

■0.0005 

•0.0037 

0.0067 

1.0000 

•0.0090 

•0.0020 

0.0009 

Coaliion  Reputation 

0.0076 

0.0057 

■0.0118 

■0.0092 

-0.0056 

•0.0122 

0.0049 

-0.0151 

•0.0075 

0.0098 

•0.0090 

1.0000 

0.0014 

•0.0211 

Coaliion  Cas  Tolerance 

•0.0003 

■0.0038 

•0.0032 

•0.0042 

•0.0000 

0.0070 

0.0039 

0.0083 

-0.0182 

0.0039 

■0.0020 

0.0014 

1.0000 

0.0099 

Coaliion  Turns  at  location 

•0.0015 

•0.0179 

-0.0130 

■0.0008 

0.0107 

•0.0177 

0.0048 

0.0026 

-0.0035 

-0.0244 

0.0009 

■0.0211 

0.0099 

1.0000 

HumanAidF  actionT  argetsandGoodDelivered 

0.0000 

■0.0000 

•0.0000 

0.0000 

0.0000 

•0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

Pair  wise  correlations  for  the  quantitative  factors  in  the  scenario  DOE.  The  largest 
absolute  value  is  .02. 
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Scatter  plot  with  factor  names  on  the  diagonal  of  the  Scenario  DOE.  This  scatter 
plot  displays  the  space  fdling  quality  of  the  NOLH  for  this  experiment. 

B.  SETTINGS  DOE 

Because  of  the  number  of  factors  and  the  number  of  design  points,  the  correlation 
matrix  for  the  setting  DOE  cannot  be  displayed  in  this  appendix.  However  the  greatest 
correlation  was  .0025,  so  clearly  our  results  qualify  as  “nearly  orthogonal”.  Below  is  the 
scatterplot  matrix.  This  figure  is  clearly  illegible.  Its  purpose  is  to  display  the  space 
filling  power  of  COL  Hernandez’s  (2008)  expansion  of  the  NOLH. 
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C.  CUMULATIVE  DOE 


[  Correlations 


Sunni  MG  Security  Sunni  Marginal  Gains  UnitFirePow  er  UnitProtection  UnitSensors  UnitSize  UnitChangeAttitudePerMan  Unitlntel 


Sunni  MG  Security 

1.0000 

-0.0001 

0.0095 

-0.0024 

0.0085 

0.0177 

-0.0172 

-0.0006 

Sunni  Marginal  Gains 

-0.0001 

1.0000 

-0.0060 

-0.0065 

0.0002 

0.0037 

0.0532 

0.0017 

UnitFirePow  er 

0.0095 

-0.0060 

1 .0000 

-0.0000 

-0.0000 

0.0061 

-0.0331 

0.0048 

UnitProtection 

-0.0024 

-0.0065 

-0.0000 

1.0000 

0.0000 

-0.0112 

-0.0167 

0.0072 

UnitSensors 

0.0085 

0.0002 

-0.0000 

0.0000 

1 .0000 

-0.0010 

-0.0029 

-0.0106 

UnitSize 

0.0177 

0.0037 

0.0061 

-0.0112 

-0.0010 

1 .0000 

-0.0127 

-0.0073 

UnitChangeAttitudePerMan 

-0.0172 

0.0532 

-0.0331 

-0.0167 

-0.0029 

-0.0127 

1.0000 

0.0102 

Unitlntel 

-0.0006 

0.0017 

0.0048 

0.0072 

-0.0106 

-0.0073 

0.0102 

1.0000 

Above  are  the  pair  wise  correlations  for  the  quantitative  factors  in  the  scenario 
DOE.  The  largest  absolute  value  is  .05. 


0.09  0.15  0.09  0.15  40  80  120  40  80  120  40  80  120  40  80  120  0  2  4  6  8  20  30  40 


This  scatter  plot  with  factor  names  on  the  diagonal 


of  the  Cumulative  DOE 


shows  the  space  filling  quality  of  this  design  of  experiment. 
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APPENDIX  E.  HUMANITARIAN  AID  SETTINGS 


In  order  to  explore  the  consent  metric,  we  gave  all  maneuver  battalions 
humanitarian  aid  providing  capabilities.  The  below  screen  shot  shows  the  attributes  we 
gave  these  units. 


Data  and  Settings  Editor 


Data  and  Settings 


Save/Load  Settings  Unit  Abilities  stances  Goods  Details  Descriptors  Population  Agents  Pre-Set  Unit  Types  Terrain  Display  Colours  Combat  Modifiers  Outputs  Weather 


Military  Values 


Unit  Type:  v 

Add  Unit 
Type 

Remove 
Unit  Type 

Output  to 

Name:  ArmdSqn(CR2) 


Reconstruction  Values 


Economic 

Sector 


Human 

Capital 


Mobility:  30 


Firepower:  80 


Change  Attitude:  3 


Protection:  TOO 


Intelligence:  20 


Social  1 

Camoflague: 
Physical  T 

Camoflague: 


Policing: 


Collateral 

Damage: 


Domain  Modifiers 


Manpower: 
Max  Footprint 


Logistics  Cost:  0 


Logistics  Provision:  0 


Save  Changes 


Income 

0.0005 

0.025 

1 

Power^r 

0.0005 

0.025 

1 

Sanj«ion 

0.0005 

0.025 

1 

Pyble  Water 

0 

0 

1 

Emication 

0 

0 

1 

fcalthcare 

0.0005 

0.025 

1 

Shelter 

0.0005 

0.025 

1 

Information 

0 

0 

1 

■ternal  Order 

0 

0 

1 

/■ministration 

0 

0 

1 

F& 

0 

0 

1 

Tra^^irt 

0 

0 

1 

Militar\ 

0 

0 

1 

Politics 

0 

0 

1 

Oil 

0 

0 

1 
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